Enterprise search has been broken for twenty years for a boring reason: keyword indexes match strings, and people ask questions. Someone types 'what is our parental leave in Germany' and gets back seventeen PDFs whose filenames contain the word 'leave'. Retrieval-augmented generation fixes the last mile of that problem — but only if the retrieval half is built properly, and that is where most implementations fail.
This is how a production RAG pipeline is actually assembled, stage by stage, and what to measure at each one.
The five stages of a RAG pipeline
Every RAG system, regardless of vendor or framework, is the same five stages. Understanding which stage is failing is most of the work of improving one.
- Ingestion — pull documents from their source systems and normalise them to text plus metadata.
- Chunking — split each document into passages small enough to retrieve precisely and large enough to be self-contained.
- Indexing — embed each chunk into a vector and store it alongside a keyword index.
- Retrieval — for a given question, fetch candidate chunks, then rerank them so the best few go forward.
- Generation — pass the question and the retrieved passages to a language model, constrained to answer only from those passages, with citations.
Stage 1: Ingestion is a permissions problem
Ingestion looks like plumbing — connect to SharePoint, Confluence, Google Drive, a ticketing system, an S3 bucket — and it mostly is. The part that is not plumbing is access control. Enterprise documents have permissions, and a RAG system that ignores them will cheerfully quote the compensation spreadsheet to an intern.
The correct design captures the source system's access-control list as chunk metadata at ingestion time, and filters retrieval by the requesting user's identity before ranking. Filtering after retrieval is not equivalent: it leaks existence and it silently degrades result quality, because the model receives fewer passages than the ranker selected.
- Store the source ACL, document owner, last-modified date and canonical URL on every chunk.
- Filter by identity as a pre-condition of the vector query, not as a post-processing step.
- Re-ingest on change rather than on a schedule — stale answers are indistinguishable from wrong ones.
- Keep a deletion path: when a source document is removed, its chunks must leave the index.
Stage 2: Chunking decides your ceiling
Chunking is the single highest-leverage decision in the pipeline and the one most often made by accepting a default. The naive approach — split every 512 tokens with a 50-token overlap — cuts tables in half, separates a heading from the paragraph it governs, and orphans the sentence that says which product the section is about.
Structure-aware chunking respects the document's own boundaries: split on headings, keep tables whole, keep code blocks whole, and prepend the heading path to each chunk so a passage carries its own context.
- Fixed-size chunking
- Split every N tokens. Fast, universal, and reliably mediocre. Acceptable for homogeneous prose, damaging for structured documents.
- Structure-aware chunking
- Split on the document's semantic boundaries — headings, sections, list groups, table units. More work per format, materially better retrieval.
- Contextual chunk headers
- Prepend the document title and heading path to each chunk's embedded text. A cheap change that consistently improves retrieval on long technical documents.
- Parent-document retrieval
- Embed small chunks for precision, but pass the larger parent section to the model for context. Best of both, at the cost of a second lookup.
Stage 3 and 4: Hybrid retrieval and reranking
Pure vector search is worse than teams expect on enterprise corpora, for a specific reason: embeddings are good at semantic similarity and bad at exact tokens. Product codes, error identifiers, version numbers, surnames and internal acronyms are exactly the terms enterprise users search for, and exactly what embeddings blur.
Hybrid retrieval runs a sparse keyword search (BM25) and a dense vector search in parallel, then fuses the ranked lists — reciprocal rank fusion is the standard, cheap and effective choice. A reranker model then scores the fused candidates against the query directly and keeps the top few.
- Retrieve broadly — 50 to 100 candidates — then rerank down to 3 to 8 passages for the prompt.
- Rerank with a cross-encoder: it reads the query and passage together and is far more accurate than embedding distance.
- Expand the query when it is short or underspecified: generate two or three paraphrases and retrieve for each.
- Log the retrieved set for every production query. You cannot debug retrieval you did not record.
Stage 5: Generation, constrained
The generation prompt has one job: answer from the supplied passages, cite them, and refuse when they are insufficient. That last clause is what separates a system people trust from one they stop using. A model that fabricates a plausible answer once will not be consulted again for anything that matters.
- Instruct explicitly that unsupported claims must be declined, and test that the refusal actually fires.
- Require inline citations to passage identifiers, and render them as links back to the source document.
- Return the retrieved passages to the interface alongside the answer, so a user can verify in one click.
- Set a groundedness check on the output: every factual sentence should be attributable to a retrieved passage.
How to evaluate a RAG pipeline
Evaluate retrieval and generation separately, because they fail differently and the fixes are unrelated. Build a golden set of 50 to 200 real questions with known correct source documents before you write pipeline code — collected from your support queue, your search logs, or an hour with the team who will use it.
- Recall@k
- Did the correct passage appear in the top k retrieved? This is the ceiling on your answer quality — generation cannot recover what retrieval never returned.
- Mean reciprocal rank
- How high did the correct passage rank? Measures reranker quality specifically.
- Groundedness
- What proportion of the answer's factual claims are supported by the retrieved passages? Catches fabrication.
- Answer relevance
- Does the answer address the question that was asked? Catches the technically-grounded non-answer.
- Refusal accuracy
- On questions your corpus genuinely cannot answer, does the system decline? The most-skipped and most-diagnostic metric.
Wire these into CI and gate merges on them. Without a regression gate, a prompt tweak that helps five questions and breaks twenty will ship, because it looked better in the demo.
What this costs to run
Cost concentrates in two places: embedding the corpus once at ingestion, and generating an answer per query. The first is a fixed, predictable one-off plus a small delta on updates. The second is what scales with usage, and where the controls are.
- Semantic caching on repeated and near-duplicate queries — in enterprise assistants, repeat rates are high, and a cache hit costs nothing.
- Model routing: send the easy majority of queries to a smaller model, reserve a frontier model for hard ones.
- Prune retrieved context aggressively — retrieved passages are usually the largest and most compressible part of the token bill.
- Better reranking reduces cost directly: fewer, better passages mean shorter prompts and better answers at once.
The short version
- Capture permissions at ingestion and filter retrieval by identity before ranking.
- Chunk on document structure, not on token count, and give each chunk its heading context.
- Use hybrid retrieval — dense plus sparse — then rerank with a cross-encoder.
- Constrain generation to the retrieved passages, require citations, and make refusal a tested behaviour.
- Build the evaluation set first, measure retrieval and generation separately, and gate CI on both.