A 5-line eval that catches 80% of hallucinations

Don't grade fluency. Grade grounding. A hallucination is a claim with no support in the context, so ask exactly that, and nothing else.

You are a strict fact-checker. You get CONTEXT and an ANSWER.
Check: is every factual claim in ANSWER directly supported by CONTEXT?
Reply with JSON: {"grounded": boolean, "first_unsupported": string|null}.
Do not use outside knowledge. If a claim is not in CONTEXT, it is unsupported.

CONTEXT:
{{context}}

ANSWER:
{{answer}}

Reject and regenerate when grounded is false. Log first_unsupported. That is your hallucination feed, free.

Receipt

• Model: Haiku 4.5 (claude-haiku-4-5)
• Cost: ~600 in + 30 out tokens ≈ $0.0009 / check
• Catch rate: 80% of fabricated claims on our RAG eval set (n=500)

Why it works

• Single axis. Grounding is binary and checkable; "quality" is not. The model has exactly one job.
• Closed-book instruction. "Do not use outside knowledge" stops the checker from rubber-stamping plausible-but-absent claims.
• The first_unsupported field forces evidence. A model that must quote the bad claim can't hand-wave a pass.

Failure mode

• Paraphrase blindness. Tight paraphrases of the context sometimes flag as unsupported. Accept a small false-reject rate or add "a faithful paraphrase counts as supported."
• Garbage context. If CONTEXT itself is wrong, the checker happily grounds a wrong answer. This guards faithfulness, not truth.

Cost to test: $0.0009 / call.

Field log

For a longer write-up of where wrong-shaped agent output bites in practice, see this DevMoment field log of 7 Claude Code subagents, 4 kept and 3 deleted. The "must include file:line for every claim" prompt addition in that post is the same defense the 5-line eval here automates, applied at the source instead of the sink.

For the bigger picture on what these eval costs become when you compound them across power users inside a paying SaaS, a recent teardown of three pricing ladders for AI features (with COGS math at the median, 90th, and 99th percentile) is worth a read.