RAG Architecture That Survives Production

Retrieval-augmented generation has a reputation for being easy, and the reputation is earned by the first two days. Load some documents, split them on a fixed character count, embed them, store the vectors, retrieve the nearest few, paste them into a prompt. A demo works within an afternoon and it is genuinely impressive, because the failure mode of a naive system on a small, clean, homogeneous corpus is subtle enough to escape a five-minute look.

The problems arrive later and they arrive from the retrieval side almost every time. Users ask questions with different vocabulary than the documents use. The corpus turns out to contain four versions of the same policy, three of them superseded. A table gets split across two chunks and the header ends up in neither. Someone asks a question whose answer requires two documents that are not similar to each other. The system responds fluently and confidently in all of these cases, because fluent and confident is what it does.

The governing constraint is simple and worth internalising before any architectural decision: the generator cannot be more correct than the material it is given. If the right passage is not in the context, no prompt engineering will recover it. Every hour spent on prompt wording while retrieval is mediocre is an hour spent on the smaller term. The work is in the retrieval pipeline, and the single decision with the widest blast radius is how you cut the documents up.

Chunking is the decision that sets your ceiling

Chunking determines what the atomic retrievable unit is, and therefore what questions are answerable at all. It is taken casually — usually as a default in a library, a fixed token count with some overlap — and then never revisited, which is how a team ends up six months later tuning a reranker to compensate for a boundary decision made in an afternoon.

The tension is exact. Small chunks give precise embeddings and tight retrieval, but arrive at the generator missing the context needed to interpret them. Large chunks carry context but produce diffuse embeddings that match everything weakly, and waste context budget on irrelevant text.

Chunk on document structure, not character counts. Headings, sections, list items, table boundaries, function definitions. A fixed-size splitter will cut through the middle of a table, separate a heading from the paragraph it governs, and split a numbered procedure across two chunks so that step four is retrievable without steps one to three. Structure-aware splitting removes an entire class of defect.

Preserve the context a chunk needs to stand alone. Prepend the document title and the heading path to each chunk. Include the section it belongs to. A chunk reading "This does not apply to customers in the Republic of Ireland" is useless in isolation and correct when it carries "Refunds Policy / Section 4: Exclusions" at the top.

Keep tables and code intact. Split a table and you have created something that looks like data and is not. If a table is too large for one chunk, repeat the header row in each piece. The same applies to code: split at function boundaries, never mid-body.

Separate what you embed from what you return. These do not have to be the same text. Embed a small precise unit; return that unit plus its surrounding neighbours, or the whole parent section. This resolves most of the small-versus-large tension directly, and it is the highest-return refinement available on a naive pipeline.

Store rich metadata alongside every chunk. Source document, version, effective date, access level, document type, section path. You need this for filtering, for citation, for freshness, and for answering "why did it retrieve that?" six weeks later.

Hybrid retrieval, because embeddings have blind spots

Dense vector search matches on meaning, which is what makes it useful and also what makes it fail in a specific, predictable way: it is weak on exact tokens. Product codes, error identifiers, surnames, version numbers, acronyms, legal clause references. These are precisely the terms users search for when they need a definite answer, and an embedding model that has learned "SKU-4471 is semantically similar to SKU-4472" will cheerfully hand you the wrong one.

Keyword search has the complementary profile. BM25 and its relatives match exact terms reliably and fail completely when the user's vocabulary differs from the corpus — asking about "time off" when the documents all say "annual leave".

Run both and combine them. Reciprocal rank fusion is the usual method and is a reasonable default: it merges ranked lists without requiring the two scoring systems to be on comparable scales, which they are not. The combination beats either alone on almost every realistic corpus, and it is a small amount of engineering.

Then rerank. Retrieval optimises for recall at speed across the whole corpus; a cross-encoder reranker scores query and candidate together and is substantially more accurate at ordering, at a cost that is only bearable over a shortlist. The standard shape is: retrieve thirty to fifty candidates by hybrid search, rerank them, pass the top handful to the generator. This two-stage structure is worth more than almost any other single improvement, because it lets you cast a wide net cheaply and then be precise about what actually enters the context.

Stage	Optimises for	Typical scope	Fails at
Dense vector search	Semantic recall	Whole corpus	Exact identifiers, rare terms
Keyword search	Exact term recall	Whole corpus	Vocabulary mismatch, paraphrase
Fusion	Combined recall	Merged candidate lists	Nothing new; it inherits both inputs' gaps
Reranking	Precision of ordering	Shortlist of tens	Cost and latency if the shortlist is large
Generation	Synthesis and phrasing	What you passed it	Anything not in the context

Metadata filtering sits across all of this and is underused. If a question is about a specific product, filter to that product before ranking rather than hoping similarity sorts it out. Access control belongs here too, applied as a hard pre-filter and never as a post-hoc instruction to the model.

Grounding and citation are architecture

An answer a user cannot verify is an answer a user should not act on, and in most commercial settings that makes citation a requirement rather than a feature.

Treat it structurally. Give each retrieved chunk a stable identifier in the context. Require the model to emit claims with references to those identifiers, using a structured output format. Then validate the references programmatically before the answer is shown: every cited identifier must exist in the context that was actually supplied. A model asked to cite will sometimes cite something plausible that was never retrieved, and this check catches it deterministically for almost no cost.

Go further where the stakes justify it. Verify that cited passages actually support the claim, either with a second model call scoring entailment or with a lighter overlap heuristic. Design the refusal path explicitly: when retrieval returns nothing above a relevance threshold, the system should say it does not know rather than synthesise from the model's parametric memory. That last behaviour is the one users find most damaging when they discover it, because it is indistinguishable in tone from a grounded answer.

The user interface carries part of this load. Link citations to the source passage, show the document version and date, and make it a single click to check. A system that makes verification easy gets trusted appropriately; a system that hides its sources gets trusted either too much or not at all.

Freshness and re-indexing

An index is a cache of a corpus that is still changing, and every cache has a staleness policy whether or not anyone wrote one down.

Build incremental indexing from the start. Full re-indexing is the thing every team does first and the thing that becomes unaffordable at exactly the point where the corpus is large enough to matter. Track content hashes per source document, re-process only what changed, and make deletion work properly — a deleted document whose chunks remain in the index is how a system confidently quotes a policy that was withdrawn last year.

Handle versions explicitly rather than by overwriting. Where superseded documents must remain accessible, tag them and filter them out of default retrieval, because otherwise your corpus contains several contradictory answers and retrieval will pick whichever is most similar to the question. Put effective dates in metadata and prefer current material in ranking.

Treat re-embedding as a migration. Changing the embedding model invalidates every vector you hold, and mixing vectors from two models in one index produces nonsense similarity scores. Plan for a parallel index and a cutover, and evaluate before you switch, not after.

Evaluate the two halves separately

This is the discipline that separates systems that improve from systems that are argued about. The single most common failure in production RAG is not a bad answer; it is a bad answer that nobody can attribute to a cause.

Measure retrieval on its own terms, with no model in the loop. Build a set of questions with the passages that answer them marked as relevant. Then compute recall at the depth you actually pass to the generator — did the right passage make it into the context at all — and the ranking quality of what came back. Recall at your context depth is the number that caps everything downstream, and it is cheap and fast to measure because it requires no generation.

Measure generation separately by giving it perfect context. Take the human-marked correct passages, pass exactly those to the generator, and score the answers. This tells you how the model performs when retrieval is not the problem.

Now the diagnosis becomes mechanical rather than a debate.

Retrieval recall	Generation with perfect context	Diagnosis
Low	High	Fix chunking, hybrid search, reranking. The model is fine
High	Low	Fix the prompt, the context assembly, or the model choice
Low	Low	Two independent problems. Fix retrieval first; it sets the ceiling
High	High, but end-to-end poor	Context assembly, truncation, or ordering is losing material

That last row deserves attention because it is the quiet one. Retrieval works, generation works, and the glue drops the best chunk because a truncation rule cut the context at a fixed length and the reranker had put the winner last. Instrument the actual prompt sent to the model, not the intended one. This is the same argument made in evaluation harnesses: a measurement without the exact configuration attached is a rumour.

What to do on Monday

Take twenty real questions from users and run them through your current pipeline. For each one, look at the retrieved chunks before looking at the answer, and record whether the passage that answers the question was present. That single number — retrieval recall at your current depth — is the most informative measurement you can take this week, and most teams have never taken it.

If recall is poor, do not touch the prompt. Look at the chunks themselves. Read fifty at random and ask whether each would make sense to a human reading it alone. Fix the boundaries, prepend heading paths, and add keyword search alongside your vector search before considering anything more sophisticated.

If recall is good and answers are still wrong, log the exact prompt sent to the model for a day and compare it to what you believe you are sending. Truncation and ordering bugs are common and invisible from the outside.

Then add citation validation as a hard check on the output path, and a refusal path for when retrieval comes back empty. Both are small pieces of deterministic code, and between them they remove the most damaging failure the system can produce: a confident, fluent, unsupported answer.