Retrieval-Augmented Generation sounds simple: embed a corpus, embed the query, fetch the top-k, stuff it in a prompt. Building one revealed that "simple" is doing a lot of work. This is the reconstruction of my five iterations on a RAG system over ~2,000 internal technical documents, with the numbers that drove each change.
The Eval First (The Part Everyone Skips)
Before changing anything, I built a gold set: 120 questions a human could answer from the docs, each with the 2–3 passages that contain the answer.
Two metrics:
- Recall@k — are the gold passages in the top-k retrieved? (retrieval quality)
- Answer correctness — judged by a strict rubric (LLM-as-judge + spot checks) (end-to-end quality)
Baseline (iteration 1): Recall@5 = 41%, correctness = 52%. The model was confidently answering from irrelevant chunks.
Iteration 1 → 2: Chunking Was the Problem
First cut: 512-token chunks, split by token count, no overlap.
Failure mode: chunks split mid-explanation. A question about "how the retry policy works" needed text that spanned a paragraph boundary my splitter ignored.
Change: structure-aware chunking — split on markdown headings first, then split long sections at paragraph boundaries, targeting 300–400 tokens with 50-token overlap.
[!NOTE]
- The lesson that stuck: chunk boundaries should follow document boundaries. Code files split by function, docs split by heading, tables stay whole. Chunking is an information-structure decision, not a token-count decision.
Iteration 2 → 3: The Embedding Upgrade
I was using a small generic embedding model to save API cost. Swapping to a stronger model (and re-indexing everything) was the single biggest jump:
Also added metadata-aware filtering: each document carries topic, service, updated_at. Queries that mention a service name filter on it before the vector search — cheaper and more precise than hoping the embedding encodes "service name."
Iteration 3 → 4: Hybrid Retrieval
Pure vector search kept missing exact terms — error codes, command names, config keys. Dense embeddings blur these into "something about configuration."
Change: hybrid search — BM25 (sparse, exact-term) + vector (dense, semantic), scores combined with reciprocal rank fusion:
bm25_hits = sparse_search(query, top_k=50)
vector_hits = dense_search(query, top_k=50)
final = reciprocal_rank_fusion(bm25_hits, vector_hits)
The exact-term misses (error codes like ERR_TIMEOUT_503) disappeared. The semantic matches that pure keyword search couldn't find stayed.
Iteration 4 → 5: Reranking
Top-k retrieval with k=5 is a knife's edge: the right passage often ranks 7th or 8th. Change: retrieve top-50, then rerank with a cross-encoder and take the top-4.
Recall@5: 93% (+8)
Latency: +380ms
The reranker is the only step that reads query and passage together — a cross-attention model, versus bi-encoder embeddings that score each side independently. That difference is the whole 8 points.
The Final Pipeline
query
→ intent + entity extraction (service names, error codes)
→ metadata filter
→ hybrid retrieval (BM25 + dense, top-50, RRF)
→ cross-encoder rerank → top-4
→ prompt with citations → answer + source links
What I'd Do Differently
- The gold set is the product. Every change was a 30-minute eval run. Without it, I was guessing; with it, I was engineering.
- Recall and correctness diverge. Iteration 3 had great recall but mediocre answers — the model was stuffing 50 chunks and getting confused. Less context, chosen better, beats more context.
- Measure latency per stage. The reranker was worth 380ms. The metadata filter saved ~90ms of vector search. Neither mattered if you don't know where the time goes.
- Chunking is versioned. When docs update, re-chunk and re-embed that document only — index invalidation is a real ops problem people hand-wave.
Numbers Summary
| Iteration | Change | Recall@5 |
|---|
| 1 | Naive 512-token chunks | 41% |
| 2 | Structure-aware chunking | 58% |
| 3 | Better embeddings + metadata filter | 74% |
| 4 | Hybrid BM25 + vector | 85% |
| 5 | Cross-encoder reranking | 93% |
RAG is not one architecture. It's five small decisions, each worth 10–17 points, and an eval harness to prove it.
Discussion (1)
Sign in or create an account to join the discussion.
2mo ago
Did you measure token cost per query after the reranking step? Curious how much it added.