Why Agentic Chats Need a Verification Step

While building an agent-powered workflow using Claude Sonnet 4.5, I ran into a frustrating pattern: the agent would claim it called tools it never invoked, fabricate facts supposedly pulled from a document, or return structured outputs that looked right but weren't grounded in reality.

The outputs were fluent, plausible, and often almost correct. But "almost correct" doesn't work when your app is supposed to reason over real documents, tools, or user data.

The fix wasn't a better prompt (although, I sure tried 😅). The fix was a verification step. This post explains why agentic chats need verification and how I implemented it.

The Failure Patterns

The hallucinations for the app clustered into a few repeatable patterns:

• Fake tool calls: The agent claimed it executed tools and referenced outputs that never existed.

• Invented document facts: It "quoted" or summarized sections of a document it never actually read or attempted to recall from compacted context.

• False confidence: The response sounded authoritative even when required steps hadn't been completed.

These failures were intermittent. They happened just enough to break trust, but not enough to be immediately obvious during development.

The Missing Layer in Most Agentic Architectures

Most agentic systems look like this:

That's fine for chat. It's not fine for systems that call tools, read files, produce structured outputs, or make factual claims users need to be able to trust.

You're basically shipping the agent's first draft directly to the user.

Agents Shouldn't Ship First Drafts

In human workflows, we review before publishing. Agentic systems should do the same.

Here's the architectural fix:

The verifier acts as a gatekeeper: it lets grounded responses through and catches hallucinations before users see them.

This single change reduced hallucinations significantly in my testing.

What the Verification Step Does

The verifier doesn't redo the task. It simply answers one question:

"Is this response making claims that can't be verified?"

It runs in three phases.

Phase 1: Skip Trivial Messages

Messages 50 characters or fewer are skipped entirely. Short messages are usually conversational, clarifying, or follow-up questions. They rarely contain tool calls or factual claims, so verifying them is wasted effort.

Phase 2: Cheap Heuristics (Regex-Based)

Most hallucinations leave fingerprints. I use simple regex checks to flag common failure patterns:

• Claimed tool usage: "I called…", "Using the tool…", "The API returned…"
• Fabricated authority: "According to the document…", "The file clearly states…"
• Structured outputs implying execution: JSON blobs referencing nonexistent IDs, ID patterns that don't match those used in the app, lists of results that were never fetched

This step is fast, cheap, and catches more issues than you'd expect.

Phase 3: Semantic Verification With a Small Model

If regex flags the message or if it's long and complex, it goes to a lightweight verifier model. I used Gemini 3 Flash for this.

The verifier prompt is intentionally narrow:

• Does this response claim tools were used?
• Are factual assertions present?
• Do those claims align with what actually happened? (i.e., were relevant tools called, is the text in the document, etc.)

The verifier doesn't rewrite anything. It just answers: safe or suspicious.

The Retry Loop

If verification fails, the response is never shown to the user. Instead, the agent gets a short corrective instruction:

"Your previous response included unverified tool calls or factual claims. Review and correct your answer, removing or fixing anything that cannot be verified."

Same agent model, second pass. The improvement is immediate: fewer invented details, more explicit uncertainty, and better alignment with actual system behavior.

Why This Works Better Than Prompting Alone

You can tell an agent "don't hallucinate," "only say what you know," or "never invent tool calls." And it will still hallucinate.

Generation-time rules are soft. Verification-time constraints are hard.

Once the model knows its output will be checked, the retry loop pushes it toward accuracy. This mirrors how humans behave under review. We're more careful when we know someone will check our work.

When You Need Verification

If your agent calls tools, reads documents, produces structured outputs, or makes factual claims users will trust, then verification isn't optional. Without it, you're shipping fluent guesses. With it, you're shipping something closer to reasoning.

Final Thoughts

Agentic chats aren't chatbots. They're first-draft machines: fast, capable, and occasionally wrong in convincing ways. And first drafts need editing. Add the verification step.