RAG systems that actually work: lessons from production

Retrieval-Augmented Generation (RAG) is everywhere right now. Every AI demo shows the same happy path: upload a PDF, ask a question, get a perfect answer. Reality is messier.

We recently built a RAG system for a legal tech client that needed to process 50,000+ case files and answer complex questions about precedent, jurisdiction, and outcomes. Here's what we learned about making RAG work in production.

The chunking problem nobody talks about

Every RAG tutorial tells you to chunk your documents into 500-token pieces with some overlap. Fine for blog posts. Terrible for legal documents where a single sentence might span three paragraphs and reference seven other cases.

We tried the standard approach: semantic chunking with embedding similarity. It worked for simple questions. It failed catastrophically for anything requiring context across multiple paragraphs. The model would retrieve the right chunks but lose the connective tissue between them.

Documents aren't just collections of tokens. They're structured arguments with dependencies, references, and logical flow. Your chunking strategy has to respect that.

Hybrid retrieval: embeddings aren't enough

Vector embeddings are great at semantic similarity but terrible at exact matches. When a lawyer searches for 'Smith v. Jones 2019', they want that exact case, not semantically similar cases about different parties.

We built a hybrid retrieval system:

• Vector search for semantic similarity (find conceptually related content)
• BM25 keyword search for exact matches (find specific case names, dates, statutes)
• Metadata filtering to narrow the search space (jurisdiction, date range, practice area)
• Re-ranking with a cross-encoder to combine the best of both

The re-ranking step was crucial. We'd retrieve 20 candidates from each method, then use a small cross-encoder model to score relevance across the combined 40 candidates. This gave us the semantic understanding of embeddings with the precision of keyword search.

Context windows aren't infinite

Modern LLMs have 128k+ context windows, so it's tempting to stuff everything in. Don't. More context means more noise, slower inference, and higher costs.

We found the sweet spot was 8-12 chunks per query, carefully selected. More than that and the model started hallucinating connections between unrelated passages. Less and it couldn't synthesize across multiple sources.

The evaluation trap

How do you know your RAG system is working? Most teams skip this and just vibe-check the outputs. That doesn't scale.

We built a proper evaluation pipeline:

• Human-curated test set of 500 question-answer pairs
• Automated retrieval evaluation (recall@k, precision@k)
• LLM-as-judge for answer quality (faithfulness, completeness, relevance)
• Regression tests to catch when changes break existing behavior

The evaluation caught bugs our manual testing missed. One change to the chunking strategy improved retrieval recall by 15% but tanked answer faithfulness—the model was retrieving more relevant chunks but mixing up which facts came from which source.

The unglamorous parts

Production RAG is 80% data engineering and 20% ML. The hard parts are:

• PDF parsing (tables, headers, footnotes all break standard parsers)
• Handling document updates (re-indexing without breaking existing citations)
• Citation tracking (showing users which chunks the answer came from)
• Latency optimization (retrieval + generation needs to be under 3 seconds)

None of this is in the tutorials. But it's the difference between a demo and a product people actually use.

The takeaway

RAG works when you treat it as a systems engineering problem, not just an ML problem. The model is the easy part. The hard part is building the retrieval pipeline, evaluation system, and data infrastructure around it.