Every RAG project I've worked on hits the same wall at week two: the demo works, but nobody can answer "is it good?" — and without that answer, every change is a leap of faith. This is the three-layer framework I've settled on. It's deliberately small: an hour to build, minutes to run, and it catches real regressions.
Layer 0: The Gold Set (30–60 Questions)
Everything depends on this. I build a set of 30–60 questions with:
{
"id": "q012",
"question": "What is the retry policy for the payment webhook?",
"answer_points": [
"exponential backoff, 5 attempts",
"dead-letter queue after final failure"
],
"gold_passages": ["docs/webhooks.md#retry-policy"],
"difficulty": "medium"
}
Rules I follow:
- Questions a real user would ask, not trivia. If the answer requires reading three docs and synthesizing, that's a hard question — tag it.
- Answer points, not full answers. Grading "does the response contain these facts" is far more reliable than "does the response match this text."
- A few unanswerable questions. "What's the refund policy for enterprise plans?" when the docs never mention it — a good system says I don't know. A bad one invents.
[!NOTE]
- 30 questions already gives you signal. I've watched a 30-question set catch a chunking regression that a 500-question "comprehensive" set (never actually run) never would have.
Layer 1: Retrieval Quality (Is the Right Context Found?)
Two metrics, both computed automatically against the gold passages:
Recall@k: what fraction of gold passages appear in the top-k retrieved?
MRR: the reciprocal rank of the first gold passage (1.0 = ranked #1)
Why separate retrieval from generation: 80% of RAG failures are retrieval failures wearing a generation costume. If the right passage never gets retrieved, no prompt will save the answer.
def retrieval_metrics(results, gold_passages, k=5):
retrieved = [r.passage_id for r in results[:k]]
hits = [p for p in gold_passages if p in retrieved]
recall = len(hits) / len(gold_passages)
first = next((i+1 for i, r in enumerate(retrieved)
if r in gold_passages), None)
mrr = 1/first if first else 0
return recall, mrr
A practical threshold I use: Recall@5 above 0.85 before touching the prompt. Below that, fix retrieval first.
Layer 2: Generation Quality (Is the Answer Right?)
I judge each answer against its answer points with an LLM judge using a strict rubric:
For each answer point:
- SUPPORTED: stated, correctly, in the answer
- MISSING: not stated
- WRONG: contradicted by the docs
Score: SUPPORTED / total points
Hard fail: any WRONG, or any hallucinated fact not in the passages
Two rules that keep the judge honest:
- The judge sees the gold passages too. It's not grading "plausible-sounding" — it's grading "supported by this evidence."
- A 5% human audit. I spot-check 3 answers per run. The judge is 95%+ aligned with me; the 5% is usually the unanswerable questions, where judges drift toward "well, it's reasonable."
Layer 3: Trust (Would I Believe This?)
The metric nobody measures until they get burned: does the system cite its sources, and do the citations hold up?
Citation faithfulness:
For each cited passage in the answer:
would the claim still be true if that passage were removed?
If yes → the citation is decorative. Penalize.
And the unanswerable-question test from Layer 0: the system should say "the documentation doesn't cover this" and stop. A RAG system that answers confidently from thin air is worse than no system, because it gets trusted.
The Dashboard (One Table, Honestly)
| Metric | What it tells you | Healthy |
|---|
| Recall@5 | Retrieval finds the context | > 0.85 |
| MRR | Context ranks high enough for the model | > 0.7 |
| Point support | Answer covers the facts | > 0.8, no WRONG |
| Hallucination rate | Invented facts | < 0.05 |
| Unanswerable rejection | Says "I don't know" when docs are silent | > 0.8 |
| Citation faithfulness | Citations actually support claims | > 0.9 |
I run this after every change — chunking, embeddings, prompt, reranker. A single number (like "accuracy") hides which layer regressed; six small numbers point at the fix.
What the Framework Caught (Real Incidents)
- Chunk overlap change: recall up 4 points, but citation faithfulness down 9 — the model cited the overlapping duplicate passage. The single-metric dashboard would have called it a win.
- Prompt rewrite: point support up, hallucination rate up 3 points. The prompt encouraged "completeness" — the model completed with invention.
- Embedding model swap: everything up except MRR, slightly. The right passage was retrieved but at rank 7 instead of 3 — the model's attention drowned in more context. Reranking fixed it.
The One-Hour Build
1. 30 questions with answer points (most of the time — worth it)
2. retrieval_metrics() (15 lines)
3. LLM judge with the rubric above (1 prompt)
4. run_eval.py: loop, score, table (an hour of glue)
RAG evaluation is not a research project. It's a test suite you refuse to ship without. The framework above is the smallest version that has caught every real regression I've had.
Discussion (0)
Sign in or create an account to join the discussion.
No comments yet. Start the discussion.