RAG in Production: The Architecture Decisions That Actually Matter

Why RAG Prototypes Fail in Production

Retrieval-augmented generation is one of the most powerful patterns for building knowledge-grounded AI applications. It is also one of the most misunderstood in terms of what it takes to make it work reliably at scale.

The pattern is straightforward on the surface: when a user submits a query, retrieve relevant context from a document store, inject that context into the prompt, and generate a response grounded in that context. In a prototype, this can be functional within a few hours.

In production, the failure modes are predictable: retrieval quality degrades for queries that are semantically ambiguous, token budgets are exceeded for documents with high information density, latency under concurrent load exceeds user tolerance, and there is no visibility into why any individual response is poor.

These are not LLM problems. They are architecture problems.

Chunking: The Foundation That Determines Everything

The quality of a RAG system is largely determined before the embedding model sees the first document. The chunking strategy — how documents are split into retrievable units — determines what the retrieval layer can and cannot find.

Fixed-size chunking (split every 512 tokens) is the default for tutorials and the wrong choice for most real documents. A 512-token chunk might split a table in half, separate a heading from its content, or combine two unrelated paragraphs that happen to be adjacent. When these chunks are retrieved and injected into a prompt, the model receives context that is artificially fragmented.

Semantic chunking preserves natural document structure: paragraphs, sections, code blocks, and tables are kept intact. Hierarchical chunking creates multiple levels of granularity — summaries at the document level, detailed passages at the section level — allowing the retrieval layer to match at the appropriate granularity for the query.

For structured documents (product catalogs, policy documents, Q&A corpora), schema-aware chunking that preserves the semantic unit of the document type outperforms generic text splitting consistently.

The right chunking strategy depends on your document corpus. There is no universal default — it requires empirical evaluation against representative queries.

Hybrid Retrieval: Dense Plus Sparse

Dense retrieval using embedding models captures semantic similarity — it can find relevant content even when the query and document use different vocabulary. But it has a known failure mode: keyword specificity. If a user asks about "PostgreSQL index bloat", dense retrieval may return documents about general database performance rather than documents that specifically discuss PostgreSQL index management.

Sparse retrieval using BM25 or TF-IDF captures lexical overlap — it excels at exact keyword matching but fails for semantic queries where the vocabulary does not match.

Production RAG systems should combine both. Hybrid retrieval using Reciprocal Rank Fusion (RRF) or linear score combination consistently outperforms either approach in isolation across diverse query types. The intuition: dense retrieval handles semantic intent, sparse retrieval handles specificity. Together they cover more of the query space.

Most modern vector databases support hybrid retrieval natively (Weaviate, Qdrant, Elasticsearch with KNN). The additional implementation cost is minimal relative to the retrieval quality improvement.

Re-ranking: The Final Quality Gate

Even with hybrid retrieval, the top-k results from the vector store are ordered by approximate similarity — not by their actual utility for the specific query. A bi-encoder embedding model, by design, produces approximate relevance rankings.

A cross-encoder re-ranker scores each retrieved document against the query jointly — producing a more accurate relevance score at the cost of higher latency (it processes each document-query pair separately, rather than comparing pre-computed embeddings).

The production pattern: retrieve top-20 candidates from the vector store, re-rank with a cross-encoder, pass the top-5 to the generation step. The re-ranking step adds 50-200ms in practice, but the improvement in context quality materially reduces hallucination rates and increases response relevance.

For latency-sensitive applications where re-ranking cost is prohibitive, a lighter alternative is metadata filtering at retrieval time — using structured attributes (document type, date, department) to reduce the candidate set before ranking.

Context Window Management

A LLM context window is a fixed resource. Filling it with marginally relevant retrieved content is a waste of that resource — and in some models, a significant driver of response degradation.

Context window management means:

• Deduplication: retrieved chunks from different documents may contain highly similar content. De-duplicate before injection.
• Relevance threshold: set a minimum similarity score below which retrieved chunks are excluded from context, regardless of the top-k setting.
• Summary injection: for queries where high-level context is more useful than raw document content, inject document-level summaries rather than detailed passages.
• Position sensitivity: models give higher implicit weight to content at the beginning and end of context. Place the most relevant retrieved content first, not appended at the end.

Inference Pipeline Architecture

The data flow of a production RAG system is not a single API call — it is a pipeline:

1. Query preprocessing (intent classification, query rewriting for retrieval, language detection)
2. Embedding generation for the processed query
3. Hybrid retrieval from the vector store
4. Re-ranking of retrieved candidates
5. Context assembly and deduplication
6. Prompt construction with retrieved context
7. LLM inference (with streaming response for UX)
8. Response post-processing and validation
9. Logging of full trace for evaluation

Each step has its own latency profile, failure mode, and optimization opportunity. Designing this as a single synchronous chain means one slow step blocks the entire response. Where steps are independent, parallelize them. Cache embedding generation for repeated or similar queries.

Observability: You Cannot Optimize What You Cannot Measure

A RAG system without observability is a system that will degrade silently. You need:

• Retrieval metrics: per-query, what was retrieved, what similarity scores were returned, did the re-ranker change the order significantly?
• Prompt metrics: token count per request, context utilization, which chunks were included?
• Generation metrics: latency, token cost, model used, finish reason
• Quality signals: user thumbs up/down, implicit signals (session continuation, follow-up questions), automated LLM-as-judge evaluation on sampled queries

These signals feed a continuous evaluation pipeline. RAG system quality is not static — it degrades when documents go stale, when query patterns shift, or when the embedding model is updated. Evaluation must be ongoing, not performed once at launch.

---

RAG architecture is not difficult — but it requires treating retrieval quality, latency budget, and observability as first-class engineering concerns, not implementation details. The systems that hold up in production are the ones that were designed with production conditions in mind from the first architectural decision.