Skip to content

00. Design a RAG Knowledge Assistant

~20 min read · Level: advanced · Part 1 of 4 (Overview → HLD → LLD → Q&A)

Problem

A retrieval-augmented generation (RAG) assistant answers natural-language questions over a private body of text the base model was never trained on. This is the product behind Glean, Notion Q&A, Microsoft Copilot over M365, and the "ask our docs" box inside a hundred enterprise tools. An employee types "what's our parental-leave policy for contractors in Germany?", and instead of the model guessing from its training, the system finds the passages in the company's own documents that actually answer the question, hands those passages to the model as context, and gets back an answer that quotes and links the real source.

The job splits cleanly in two, and keeping them separate is most of the design. There is an offline ingestion side that turns a corpus of documents into a searchable index of embedded text chunks, and an online query side that, per question, retrieves the most relevant chunks, reranks them, feeds the best few to the model, and returns a grounded, cited answer. The model is the cheap, interchangeable part. The hard engineering is getting the right few thousand tokens of context in front of it — because the model can only answer from what retrieval surfaces, and it will answer confidently whether or not the right passage was found.

To keep the reasoning concrete, thread one scenario through the whole design: an enterprise assistant deployed over 10,000,000 internal documents — wikis, design docs, tickets, policy PDFs, support threads — used by roughly 50,000 employees. Two properties of this deployment stress every decision below. First, every answer must cite its sources: the response links the exact documents the claim came from, and a user who can't verify an answer won't trust it. Second, the index must stay fresh: when someone edits the parental-leave policy and saves it, a colleague asking about it five minutes later must get the new version, cited to the new revision — not yesterday's snapshot. Ten million documents, mandatory citations, and a five-minute freshness window: those three constraints will test every choice from chunk size to index topology.

Functional requirements

  • Ingest documents from source connectors (wiki, docs, ticketing, drive), split each into chunks, embed them, and index the vectors plus metadata.
  • Answer a natural-language question grounded in the corpus, returning synthesized prose with inline citations that resolve to the exact source document and passage.
  • Enforce access control: a user's answer may only draw on documents that user is permitted to see. Permission is filtered at retrieval time, not patched on after generation.
  • Stay fresh: a document edited now is reflected in answers within minutes, without rebuilding the whole index.
  • Abstain: when the corpus does not contain the answer, say so ("I don't have a source for that") rather than fabricate one.

De-scoped for this round, and worth naming so the interviewer sees it as a choice rather than an oversight: multi-hop agentic reasoning and tool use beyond retrieval, fine-tuning or training the generation model, cross-session conversational memory, and document authoring/editing. These are real product surfaces, but they sit beside the core retrieve-then-generate loop and do not change its architecture.

Non-functional requirements

The single dominant constraint is answer groundedness, gated by retrieval recall under a bounded context window. Everything downstream follows from it. The model can synthesize only from the chunks retrieval hands it, and the context window caps how many chunks fit — so if the passage that answers the question is not in the top-k that retrieval surfaces, no amount of model quality recovers it. The system's correctness lives or dies on retrieval recall, not on the eloquence of the generator.

  • Groundedness: every factual claim in an answer must trace to a retrieved chunk; citations must resolve; the system abstains when confidence is low. This is the property users actually judge.
  • Freshness: edit-to-queryable latency under 5 minutes (the scenario), without a full reindex.
  • Latency: p95 to first token under ~1.5 s, full answer under ~4 s. Generation dominates this budget, which — as we'll see — is what lets us spend generously on retrieval quality.
  • Security: access control enforced at retrieval and failing closed. A permission bug here leaks confidential documents through a chat box, the worst failure the system can have.
  • Availability: the query path stays up; ingestion can lag briefly without taking answers down.

Scale estimation

Start from the corpus. 10M documents, averaging ~3,000 tokens each (a mix of short tickets and long policy PDFs), is roughly 10M × 3,000 = 30 billion tokens of source text.

Chunking drives everything else. Split each document into 512-token chunks with 64-token overlap, so the effective stride is ~448 tokens and a 3,000-token document yields about 3000 / 448 ≈ 6.7, call it ~6 chunks per document. Across the corpus that is 10M × 6 = 60 million chunks — 60M is the number the whole index is sized around.

Each chunk becomes one embedding vector of 1,024 dimensions. At float32 (4 bytes/dim) a vector is 1,024 × 4 = ~4 KB, so the raw vectors are 60M × 4 KB = 240 GB, and an HNSW graph adds roughly 1.5× overhead for its links → ~360 GB of RAM to hold the index at full precision. That is the cost lever: apply int8 scalar quantization and each vector drops to ~1 KB, so the index is 60M × 1 KB = 60 GB — small enough to hold in memory across a handful of nodes, at a small, measurable recall cost we revisit in the tradeoffs.

The chunk text lives in a separate store, not in the vector index: 60M chunks at ~2 KB of text plus offsets each is ~120 GB on cheap disk/blob. The vector index holds only vectors plus a few filter fields; the bulky text is fetched by id after search.

Query load is modest in QPS but expensive per query. 50,000 employees doing ~4 assistant queries a day is ~200,000 queries/day, or 200,000 / 86,400 ≈ 2.3 QPS average, ~23 QPS at peak. The ANN search itself shrugs at 23 QPS over 60M vectors. The cost is downstream: each query fires one query-embedding, one vector search, one cross-encoder rerank of ~100 candidates, and one LLM generation over ~4,000 tokens of assembled context. Generation is the expensive, GPU-bound step; retrieval is cheap by comparison, and that asymmetry shapes the whole design.

Ingestion cost splits into a one-time build and a steady trickle. The initial build embeds all 30B tokens; at ~1,000 chunks/s on one embedding GPU that is 60M / 1,000 = 60,000 s ≈ 16.7 hours, so parallelize across ~20 GPUs to finish in under an hour. The steady state is tiny: one edited document is ~6 chunks (~3,000 tokens) to re-embed, a few hundred milliseconds and a fraction of a cent — which is exactly why the five-minute freshness window is affordable and a nightly full rebuild is not.

API sketch

POST /api/v1/query
  body: { "question": "...", "user_id": "...", "top_k": 8, "filters": {...} }
  200:  { "answer": "Contractors in Germany accrue leave under [1]...",
          "citations": [ { "doc_id": 42, "version": 7, "title": "...",
                           "url": "...", "snippet": "...", "score": 0.83 } ],
          "confidence": 0.83 }
  204:  abstained (no grounded answer available)

POST /api/v1/documents            # ingest or upsert one document
  body: { "doc_id", "source", "content" | "uri", "acl": [...], "version", "metadata" }
  202:  { "doc_id", "status": "queued", "chunks_estimated": 6 }

DELETE /api/v1/documents/{doc_id} # tombstone + purge from index
  202

GET /api/v1/documents/{doc_id}/status
  200:  { "indexed_at", "version", "chunk_count", "index_version" }

Solutioning

Start from the dominant constraint and the shape falls out. Because correctness is a retrieval problem, the system is built as a two-stage retriever feeding a generator, not as "a prompt wrapped around an LLM." Stage one is fast and cheap: embed the question and pull the top ~100 candidate chunks from an approximate-nearest-neighbor index. Stage two is slower and sharper: a cross-encoder reranker rescores those 100 against the question and keeps the best ~8, which get assembled into ~4,000 tokens of context for the model. The single most useful reframing to carry into the room: a wrong RAG answer is usually not a generation problem; it's a retrieval-recall problem — the fix is almost never a better prompt and almost always getting the right chunk into the top-k.

The first defining tradeoff is chunk size versus recall, and it moves real numbers. Small 256-token chunks pinpoint a specific fact but double the index to ~120M vectors and fragment context, so a claim that spans two chunks gets split and neither ranks well. Large 1,024-token chunks halve the index to ~30M vectors and keep context intact, but each chunk dilutes the one relevant sentence among a thousand tokens, dropping the recall of pinpoint facts and letting fewer distinct chunks fit the context window. 512 tokens with 64-token overlap is the resolution: overlap stops a fact from being guillotined at a boundary, and the size keeps chunks topically tight without exploding the index. Where chunks still lose their bearings — a chunk that says "the limit is 30 days" without saying what limit — prepend a short contextual prefix (document title and section) at ingestion so the chunk carries its own context.

The second tradeoff is retrieval latency versus quality, and here the budget makes the call easy. The reranker over 100 candidates costs ~200 ms; the ANN search ~30 ms; the query embedding ~40 ms. Against a generation step that runs 1.5–2.5 seconds, that entire retrieval stack is under 10% of the latency budget. So we spend generously on retrieval — a heavy cross-encoder reranker, an over-fetch of 100 candidates — because quality there is nearly free relative to the generation we're already paying for. This is not "retrieval must be fast"; it's "retrieval is cheap, so make it good."

The third tradeoff is index freshness versus cost, and it is the crux of the five-minute scenario. Re-embedding all 60M chunks nightly costs ~16 GPU-hours and reflects an edit only the next day — wrong for a policy that changed this morning. The resolution is incremental upsert on change events: a connector emits an edit, an ingestion worker re-chunks and re-embeds only that document's ~6 chunks, and upserts them into the live index. Freshness here is not a full-reindex problem; it's an incremental-upsert problem — the edited document is queryable in minutes at the cost of six chunks, not thirty billion tokens.

The last decision is hallucination control, which is the direct expression of the groundedness constraint. Three levers stack: pass the model only the retrieved chunks and instruct it to answer solely from them; require inline citations and verify each one resolves to a real chunk before returning; and abstain when the top reranked score falls below a confidence threshold, returning "I don't have a source for that" instead of a fluent guess. Access control rides the same retrieval stage — the ANN search filters candidates by the user's permissions and fails closed, so a document the user can't see can never reach the model's context, let alone its answer. The following files take each of these decisions down to components (HLD) and then to schemas, algorithms, and edge cases (LLD).