Interrogating AI’s Blind Spots
What happens when we treat ChatGPT, Claude, or another AI platform’s first answer as the complete answer, without checking what it left out? This post explores why AI only answers what it’s asked, and why interrogating those answers is still, and always will be, a human responsibility.
Listen to “Interrogating AI’s Blind Spots” (Audio Deep Dive)
As I worked on a client project this week, part of my analysis was to understand how ChatGPT, Perplexity, Claude and Gemini’s search results differed, and where the client and its services were being cited, or not.
Informed by the client’s website, competitors and industry data, I drafted prompt variants that a prospective customer would ask. I ran my ideas through Claude, which proceeded to refine the prompts I would use for my AI visibility audit. At my request, Claude also generated a few additional prompts to test against, built around what I might have missed. While they were all good, with no hallucinations, I picked up on several areas of interest that I already knew from the client’s website that Claude’s prompts had missed entirely.
I proceeded to “interrogate” Claude to understand how it arrived at answers that completely missed what I’d already seen on the site. It reminded me that AI works from what it’s given. Without a source to check, like a website, and a specific enough question, it can’t tell you what it missed.
By going back to Claude, not accepting the first pass and asking for more context about the phrasing of the prompts, I was able to conduct a much more thorough audit that covered multiple audiences, budgets, services the client offered and a few outlier questions.
“What happens when we treat AI’s first answer as the whole answer? And what’s the cost of accepting that answer without checking what it left out?”
What History Teaches Us
History offers a similar parallel about the importance of questioning what technology tells us. The 1979 Three Mile Island nuclear accident was caused by a misleading interface.
When a relief valve became stuck open, the control panel light turned off, signaling that power to the valve had been cut. The power plant’s operators took the unlit lamp to mean the valve was physically closed. Trusting the light over the valve itself, operators reduced flow to the emergency cooling system, which turned an equipment failure into the worst commercial nuclear power accident in U.S. history.
The lesson isn’t that the light lied; rather, it’s that everyone assumed a working light meant a working system. We should be checking whether other evidence confirms it, or contradicts it, entirely.
With AI, Question Everything
This KPMG and University of Melbourne study found that 66% of the workforce relies on AI output without evaluating its accuracy. The study emphasizes the importance of constant human oversight with AI, especially against unintended consequences.
What about when you look to operationalize AI without proper guardrails and procedures? Anthropic’s Project Vend, which we covered here, tested whether an AI agent could run a vending machine business autonomously. Within weeks, it had given away inventory for free, ordered a PlayStation 5 and lost over $1,000.
Yet, it’s easy to see how this happens. Most of us read AI’s first answer as complete because it sounds thorough, confident and credible. Whether it’s a control room panel in 1979 or an AI search result today, when we allow our judgment to be shaped by what we assume has been vetted and verified, we put our own reputation, credibility and sometimes safety at risk.
The industry has a name for this: automation bias. First identified by psychologists studying aviation crews in the 1990s, it describes the tendency to treat an automated system’s output as a substitute for our own judgment, and it’s exactly what’s happening now, every time we accept AI’s first answer as the complete answer.
Governing AI the Same Way You Would Your Own Team
When working with AI, think of it as defining clear roles and responsibilities, which isn’t so different from delegating tasks to individuals on your team. You assign each person a role in researching or exploring the adjacencies of a question. As their supervisor or team lead, it’s still your responsibility to challenge their answers—i.e., how they arrived at a conclusion and what opportunities or lessons the adjacent findings reveal.
It’s no different with AI, where the speed of answers is almost immediate, and the answers it provides are limited to only the direction you give it. The responsibility still sits solely with you to verify everything, to look beyond what the model knows in order to understand where its grasp of the audience or use case falls short. Think of a traditional library built on index cards and books: the information is all there, we still have to sift through it and verify what we find and why it’s the right information.
This AI Slop Survey of U.S. workers found that more than one in three rarely or only occasionally review AI-generated output before using it, and 18% say they usually trust it as is.
As McKinsey puts it, “Agents and robots can drill into data sets to identify patterns and generate options.” But interpreting what those patterns mean, and deciding what to do next, is still a human responsibility.
The idea of “interrogating” AI needs to be incorporated into a standard set of AI safety procedures that individuals, companies and boards adhere to. Question AI the same way you’d question your team about how they arrived at their answers—not just to validate a point of view, but to expand on it.
What Have AI Search Audits Taught Me?
AI can convincingly role-play the audiences a business is trying to reach, but it still can’t tell you what a website or what searchable content is missing on its own. I’m fortunate enough to understand that a client’s website is much more than simply user experience, design and technical needs. AI, on the other hand, conducts its own first pass in a fraction of a second, sourcing that same website for answers. Will it look at the site through the eyes of the business, or the customer’s own needs and perceptions? What about the internal team’s sales and revenue goals? At my direction, I can instruct AI to play the role of each of these audiences, and surface gaps, white spaces and competitive pros and cons.
The findings AI generates still require a human to decide what to do with them. But what happens once you have that feedback? How does it compare to the information that still needs to be surfaced within the organization, deciphered and activated? That’s something AI can’t do, and it’s where a company’s limited depth of knowledge starts working against it.
Across roughly a dozen AI visibility audits, the clearest pattern isn’t which company gets cited; rather, it’s that every AI platform interprets the same prompt differently. ChatGPT cited one company cleanly across five prompts, while Claude and Perplexity cited the same company only twice. Sometimes the website leads the citations; other times, it’s invisible. That inconsistency is a clear demonstration of why third-party sites and public relations (PR) are table stakes in winning the AI citation battle.
Ask yourself: “What’s the one answer from ChatGPT, Claude, or another AI tool that you assumed was thorough, and later realized wasn’t?”
Beyond that, the outcome depends on the industry, product, service and geography. What’s most surprising is the gap between what a client assumes they’ll be cited for and what AI actually decides to cite them for. Some audits have exposed real business opportunities to be cited against. Others point to unrealized revenue growth through audiences a company and its website are currently invisible to.
It’s proof that we, not AI, are the ones who still need to ask the question nobody has asked yet: What’s the one answer from ChatGPT, Claude, or another AI tool that you assumed was thorough, and later realized wasn’t?
If you’re interested in seeing how AI currently cites your company, and what blind spots may exist when customers search for your company or service, I’d welcome the chance to show you.
For more on AI strategy and its business impact, or to continue the conversation, let’s talk. Email me at mike@scheinerinc.com.
As always, thank you for reading and/or listening.
