Production RAG that doesn't hallucinate

Key takeaways

• When a RAG system hallucinates, look at what was retrieved before blaming the model. It's the retrieval, most of the time.
• Chunk along the document's own structure, not by token count. A chunk that ends mid-sentence retrieves like noise.
• Hybrid retrieval (keyword + vector, then rerank) is not an optimization — for real corpora with names, codes, and jargon, it's the baseline.
• Give the model permission to say "not in the documents." A system that must always answer will always eventually lie.
• Evaluate retrieval separately from generation, with a golden set, before touching a single prompt.

The uncomfortable diagnosis

A RAG demo takes an afternoon. A RAG system that a team trusts with their own documents takes longer, and the gap is almost never the language model. In the systems we've built and the ones we've been called in to fix, the chain of failure is usually:

1. The answer's source passage was never retrieved, or arrived shredded by bad chunking.
2. The model, handed weak context and an instruction to be helpful, does what it does best: writes something plausible.
3. The user reads a confident, cited-looking answer that the documents do not support.

Step 2 gets the blame. Steps 1 and 3 are where the engineering lives.

Chunking: respect the document

Fixed-size token chunking is the default in every tutorial and the first thing we rip out. Documents have structure — headings, clauses, tables, procedures — and retrieval quality tracks how well chunks preserve it.

What works in practice:

• Split on structure first (headings, sections, list boundaries), size limits second. A slightly oversized coherent section beats two fragments.
• Prepend a breadcrumb to every chunk: document title and heading path. Employment Agreement > Termination > Notice periods turns an orphaned paragraph into something both retrievable and citable.
• Tables are not prose. Extract them separately and describe them, or query them structurally. A table flattened into a sentence stream is destroyed information.
• Keep the source anchor — page, section, character offsets — on every chunk. Grounding depends on it later.

Retrieval: hybrid or you will get burned

Vector search is magnificent at paraphrase and hopeless at exactness. Real questions are full of exactness: part numbers, statute references, error codes, people's names. Pure embedding search will happily return "something about invoices" when the user asked about invoice INV-2209.

The production baseline:

query → BM25 (keyword)  ─┐
                          ├→ merge → cross-encoder rerank → top 5-8 chunks
query → vector search   ─┘

BM25 catches the exact tokens, vectors catch the paraphrase, and the reranker — which sees the query and chunk together — decides what actually answers the question. On every corpus we've measured, the reranker step moves answer quality more than any prompt change.

Retrieve generously (30–50 candidates), rerank hard, pass few. Long context windows tempt you to dump everything in; relevance beats volume, and the model's attention is a budget like any other.

Grounding: make lying structurally hard

The generation prompt matters less than its constraints. Three that pay for themselves:

• Quote-then-answer. The model first extracts verbatim spans it will rely on, then composes from those spans. Skipping the quote step is where invention sneaks in.
• Citations as data, not decoration. Every claim maps to a chunk ID your UI can resolve to the exact passage. If a sentence has no source chunk, it doesn't ship.
• A dignified "not found." If reranked scores are weak, the correct output is "the documents don't cover this" plus the nearest related passages. Systems forbidden from abstaining will hallucinate exactly when it matters most.

Evals: retrieval first, always

Teams tune prompts against vibes because prompts are easy to change. Do the boring thing instead:

1. Build a golden set — 50 to 150 real questions with hand-verified source passages and answers. Painful, worth it.
2. Measure retrieval alone: is the right passage in the top k? If recall@8 is 60%, no prompt on earth saves you.
3. Only then evaluate generation: faithfulness to sources, and correctness against the golden answers.
4. Re-run on every corpus change, not just code change. New documents shift retrieval behavior in ways nobody predicts.

Symptom	Actual cause, usually	Fix
Confident wrong answers	Source never retrieved	Hybrid retrieval, chunk repair
Right document, wrong detail	Chunks shredded mid-thought	Structural chunking, breadcrumbs
Misses exact identifiers	Vector-only search	Add BM25 + rerank
Answers drift from sources	No quote step, no citation contract	Grounded generation
Great in staging, bad live	Eval set doesn't match real queries	Mine golden set from production

Where this connects

Retrieval is one leg of the systems we build — it's the "memory" a pipeline reaches for before tools and agents get involved. When multiple agents share that memory and check each other's use of it, you're in orchestration territory: Planner, critic, verifier. And when the ambiguous step lives inside an otherwise deterministic workflow, start smaller: When your n8n automation needs a brain.

FAQ

Do long-context models make RAG obsolete? No — they change the economics of k, not the need for retrieval. Feeding a million tokens to answer a question about one clause is slow, expensive, and measurably noisier than five reranked chunks. Context windows are a budget; retrieval is how you spend it well.

Which embedding model should I pick? It matters less than chunking and reranking. Pick a competent recent model, then spend the week you saved building the golden set — that's the asset that compounds.

Knowledge graphs or vectors? Not either/or. Graphs shine when questions traverse relationships ("which contracts share this clause"); vectors shine at "find me the passage." Our document-intelligence work usually ends up with both: a graph for structure, embeddings for reach.

---

This is the pattern library behind our Knowledge Retrieval and Document Intelligence modules. If your RAG system is confidently wrong, the door is open.