01. RAG Knowledge Assistant — High-Level Design¶

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the solutioning narrative into concrete boxes and flows. The system has two distinct paths — an offline ingestion path that builds the index and an online query path that answers questions — so read the diagram as two pipelines meeting at the vector index, then follow a document through ingestion and a question through query, then look at what breaks.

Architecture¶

  INGESTION PATH (offline / streaming)
  ┌───────────┐   ┌───────────────┐   ┌──────────────┐   ┌────────────┐
  │ Source     │   │ Change-event  │   │ Ingestion    │   │ Chunker    │
  │ connectors │──▶│ queue (Kafka) │──▶│ workers      │──▶│ + context  │
  │ wiki/jira/ │   │               │   │ (dedupe,     │   │ prefixer   │
  │ drive/pdf  │   └───────────────┘   │  version)    │   └─────┬──────┘
  └───────────┘                        └──────────────┘         │
                                                                ▼
                                                        ┌──────────────┐
                                                        │ Embedder GPU │
                                                        │ (model vN)   │
                                                        └──────┬───────┘
                             upsert vectors + delete orphans   │
        ┌───────────────────────────────────────────────┬─────┴─────────┐
        ▼                          ▼                     ▼               ▼
 ┌──────────────┐         ┌──────────────┐      ┌──────────────┐  ┌──────────────┐
 │ Vector index │         │ Chunk store  │      │ Document /   │  │ (index has   │
 │ ANN, int8,   │         │ (KV: id→text │      │ metadata +   │  │  minimal     │
 │ sharded      │         │  + offsets)  │      │ ACL store)   │  │  filter tags)│
 └──────▲───────┘         └──────▲───────┘      └──────▲───────┘  └──────────────┘
        │                        │                     │
  ══════╪════════════════════════╪═════════════════════╪═════════════════════════
        │  QUERY PATH (online)    │  fetch text by id   │  ACL lookup
        │                        │                     │
 ┌──────┴───────┐   ┌───────────┐│┌───────────┐ ┌──────┴──────┐ ┌──────────────┐
 │ Query        │──▶│ Query     ├┼┤ ANN search│▶│ Reranker    │▶│ LLM          │
 │ orchestrator │   │ embedder  │││ + ACL     │ │ cross-encdr │ │ generator    │
 │  (per query) │   │ (model vN)│││ filter    │ │ 100 → 8     │ │ (+ citation  │
 └──────▲───────┘   └───────────┘│└───────────┘ └─────────────┘ │  verifier)   │
        │  question / answer      │                              └──────┬───────┘
   ┌────┴────┐                    │  top-100 ids                        │ grounded
   │ Client  │◀───────────────────┴─────────────────────────────────────┘ answer
   └─────────┘                                                    + citations

Read it as two pipelines sharing three stores. Top-down on the left is ingestion: connectors watch the source systems and emit change events onto a durable queue; workers dedupe and version them, chunk each document and prefix each chunk with its context, embed the chunks on GPU, and upsert the vectors into the index while writing chunk text to the chunk store and document facts (version, ACL, hashes) to the metadata store. Left-to-right in the lower band is query: the orchestrator embeds the question with the same model version the index was built with, runs an ACL-filtered ANN search for ~100 candidates, fetches their text, reranks to ~8, and hands those to the generator, which produces a cited answer and verifies each citation before returning it.

Components¶

Source connectors. One adapter per source system (Confluence, Jira, Google Drive, a PDF crawler). Their job is change capture: detect creates, edits, and deletes and emit a normalized change event with the document id, source version, ACL, and either the content or a URI to fetch it. Doing capture here — rather than polling the whole corpus — is what makes incremental freshness possible.

Change-event queue. A durable log (Kafka or equivalent) between capture and processing. It decouples bursty source edits from bounded embedding throughput, gives replay for reprocessing after a bug, and provides a dead-letter queue for documents that fail to parse. Its depth is the single best proxy for freshness lag.

Ingestion workers. Stateless consumers that own dedupe (skip re-embedding a document whose content hash is unchanged), versioning (ignore an edit older than what's indexed), chunking, and orphan cleanup (delete chunks a shrunk document no longer has). They call the embedder and write to the three stores transactionally-enough that the index never points at missing text.

Chunker + context prefixer. Splits documents into 512-token, 64-overlap chunks on sensible boundaries (headings, paragraphs) and prepends each chunk with a short context line — document title and section path — so a chunk reading "the limit is 30 days" carries what limit and from where. This directly lifts recall on chunks that would otherwise be unmoored.

Embedder (GPU, versioned). Turns chunk text into 1,024-dim vectors. It is version-pinned: the model that embeds documents at index-build time must be the exact model that embeds queries, or the distances are meaningless. The version tag travels with every vector and every query.

Vector index (ANN). The in-memory approximate-nearest-neighbor index over 60M int8 vectors, sharded across nodes. It holds vectors plus a few filter fields (doc_id, ACL tag, index_version) and nothing bulky. Its one hot operation is "given a query vector and an ACL filter, return the top-100 nearest chunk ids."

Chunk store. A key-value store mapping chunk_id → chunk text, source offsets, and doc_id. The vector index returns ids; this store turns ids back into the text the generator reads and the citation resolves to. Kept separate so the index stays small and RAM-resident.

Document / metadata + ACL store. A relational or document DB keyed by doc_id holding source, monotonic version, content hash, ACL principals, indexed_at, index_version, and tombstone status. This is the transactional record that drives dedupe, version ordering, and permission filtering.

Query orchestrator. The per-query coordinator: embed → search → fetch → rerank → assemble → generate → verify. It owns the latency budget, the abstain decision, timeouts, and graceful degradation when a downstream stage is slow.

Reranker (cross-encoder). A GPU model that scores each (question, candidate-chunk) pair jointly — far more accurate than the bi-encoder ANN score, because it reads the question and chunk together. It reorders the 100 candidates and keeps the top ~8. It is where most of the answer-quality lift comes from, and it is cheap against the generation budget.

LLM generator + citation verifier. Assembles the top chunks into a numbered context block, prompts the model to answer only from that context with inline [n] citations, streams the answer, and verifies that each cited marker maps to a real chunk before the answer leaves the building. If verification fails or the retrieval confidence was low, it abstains.

Primary write path (ingest / update a document)¶

A connector detects an edit to document 42 and emits a change event {doc_id:42, version:7, op:upsert, acl:[...], content|uri} onto the queue.
An ingestion worker pulls it and reads the current record from the metadata store. If the incoming version is not newer than what's stored, it drops the event (stale). If the content hash matches what's indexed, it skips re-embedding entirely and just updates metadata.
Otherwise the chunker splits the new text into ~6 chunks and prefixes each with the document's title/section context.
The embedder turns the 6 chunk texts into 6 vectors with model version vN.
The worker upserts the 6 (chunk_id, vector, doc_id, acl_tag, index_version) entries into the vector index and writes their text to the chunk store, then deletes orphaned chunks the previous revision had but this one doesn't.
It updates the document record — version 7, new content hash, indexed_at = now, chunk_count — which is the commit point that makes the edit officially queryable.

Primary read path (answer a question)¶

POST /api/v1/query reaches the orchestrator with the question and the user id.
The orchestrator embeds the question with model vN (the index's version) — ~40 ms.
It runs an ANN search for the top ~100 nearest chunks, filtered by the user's ACL principals so nothing the user can't see is a candidate — ~30 ms. The filter fails closed: if ACL data is missing, the chunk is excluded.
It fetches the 100 candidates' text from the chunk store by id.
The reranker scores all 100 (question, chunk) pairs and keeps the top ~8 — ~200 ms.
If the top reranked score is below the confidence threshold, the orchestrator abstains (HTTP 204) — no fabricated answer.
Otherwise it assembles the ~8 chunks (~4,000 tokens) into a numbered context block and streams generation from the LLM — 1.5–2.5 s.
The citation verifier confirms each [n] marker resolves to a real chunk, attaches the source metadata (doc_id, version, url, snippet), and returns the grounded answer with citations.

Storage choices¶

Vector index: in-memory ANN (HNSW-class), int8-quantized, sharded. The access pattern is a single approximate top-k query per request; there are no joins and no scans. Quantization cuts the footprint from ~360 GB to ~60 GB so the whole thing stays RAM-resident for ~30 ms search, at a small recall cost the reranker then repairs. Metadata on each vector is kept minimal (ids and filter tags) precisely so the index stays small.
Chunk text: key-value store. Point lookups by chunk_id, high fan-out (fetch 100 per query), no relational shape — a KV store or blob with an id index fits. Keeping the bulky text out of the vector index is what lets the index be small and fast.
Document metadata + ACL: relational / document DB. Needs transactional version updates, content-hash comparisons, and permission queries — a system of record, not a cache. This is the store that arbitrates freshness and security, the two places you cannot be sloppy.
Semantic answer cache (optional): KV keyed by normalized question. Popular repeated questions ("what's the VPN setup?") can serve a cached answer, invalidated when any cited document's version or ACL changes. It trims generation cost on the head of the distribution without risking staleness on the tail.

Scaling¶

Query path. ANN search over 60M vectors at 23 QPS peak is comfortable on a few sharded nodes; scale it by sharding the index (each shard searches its slice, a coordinator merges top-k) and adding replicas for QPS and availability. The real capacity limits are the GPU stages: the reranker scores ~100 pairs/query, so 23 QPS × 100 = ~2,300 pairs/s — one GPU handles that, and you add replicas linearly. Generation is the scarce resource; it is queued and autoscaled across a GPU pool (or fronted by a hosted model with a concurrency budget), and the semantic cache shaves the repeated head of the query distribution off that pool.

Ingestion path. The initial build is embarrassingly parallel: 60M chunks at ~1,000 chunks/s/GPU is 16.7 GPU-hours, so 20 embedding GPUs finish in ~50 minutes. Steady state is trivial — edits trickle in as ~6-chunk jobs — and the queue absorbs bursts (a bulk import of 100k documents) so the online path never feels ingestion pressure. Scale ingestion by adding queue consumers and embedder replicas independently of the query tier.

Index growth. Doubling the corpus to 20M documents doubles vectors to 120M and int8 index memory to ~120 GB — add shards, don't grow a node. The chunk store grows to ~240 GB of cheap disk, a non-event. The number that actually needs watching as the corpus grows is not storage but recall: more chunks means more near-duplicates competing for the top-k, which is a reranking-and-dedup problem, not a capacity problem.

Operational signals¶

The healthy signal is edit-to-queryable latency sitting flat under the 5-minute SLO and the citation-resolution rate near 100% — answers reliably point at chunks that exist. The first metric to degrade under trouble is the change-queue depth / ingestion lag: when it climbs, freshness silently slips and the assistant starts citing yesterday's revision of a document edited this morning, long before any error fires. The misleading metric is GPU utilization and token-throughput on the generator — they can look perfectly healthy while retrieval quietly returns irrelevant chunks and the model fluently hallucinates over them; a green generation dashboard says nothing about whether the right passage was found. The graph an experienced operator opens first during a quality incident is the retrieval-score distribution alongside the abstain rate: a sudden drop in top reranked scores (or a spike in low-confidence abstains) localizes the fault to retrieval — a bad reindex, a model-version mismatch, or a filter dropping candidates — rather than to the generator everyone instinctively blames.

Failure modes and resilience¶

Ingestion lag → stale citations. If the change queue backs up (a bulk import, an embedder outage), a document edited 5 minutes ago isn't indexed and the assistant answers from the old revision — silently, with a confident citation to stale text. This is the scenario's core failure. Mitigations: alert on queue depth and edit-to-queryable p95, prioritize small interactive edits ahead of bulk backfills in the queue, and expose indexed_at/version in citations so a reader can see how fresh the source is.
Embedding-model version skew. If queries are embedded with model vN+1 while the index holds vN vectors, the nearest neighbors are garbage and answers degrade without any error. Mitigation: pin the version tag on every vector and every query, refuse to search across versions, and reindex behind a dual-index cutover when upgrading the model.
ACL filter bug → data leak. The worst failure: a document a user shouldn't see reaches their answer. Mitigation: enforce the permission filter inside the ANN search (not as a post-filter that can be bypassed), fail closed on missing ACL data, and delete a tombstoned document's vectors immediately rather than lazily.
Vector index node loss. A shard going down drops recall for the chunks it held. Mitigation: replicate each shard so a coordinator can read from a replica, and accept briefly degraded recall (the reranker still orders what's returned) rather than failing the query.
LLM outage or timeout. Generation is the least reliable, highest-latency stage. Mitigation: on timeout, degrade gracefully — return the top reranked citations as "here are the most relevant sources" without synthesized prose — so the user still gets verified links instead of an error.
Near-duplicate poisoning. Copied-around documents fill the top-k with eight paraphrases of one passage, starving the context of diversity. Mitigation: dedup by content hash at ingestion and apply diversity (MMR) at rerank so the 8 slots hold 8 distinct sources.

Where this shows up in production¶

Glean — permission-aware enterprise search where the ACL filter runs inside retrieval per user, so two employees asking the same question get answers trimmed to what each may see — exactly the fail-closed retrieval-time filter here.
Microsoft 365 Copilot — retrieval over the Graph is permission-trimmed before anything reaches the model, the canonical "security lives at retrieval, not generation" pattern.
Perplexity — retrieve, rerank, and generate with mandatory inline citations and an abstain discipline, the same grounded-answer contract this design commits to.
Notion Q&A — RAG over a workspace with incremental re-index on edit, the incremental-upsert freshness path rather than a nightly rebuild.
Anthropic contextual retrieval — prepending document/section context to each chunk before embedding to fix the "unmoored chunk" recall loss, the role the context prefixer plays here.
Cohere Rerank / cross-encoder rerankers — the productized second stage that rescores a cheap first-stage candidate set, the retrieve-then-rerank split this design leans on.
Pinecone / Weaviate / pgvector — ANN indexes with metadata filtering and live upserts, the vector-index role including the ACL-filter-and-upsert operations.
Elastic / OpenSearch hybrid search — BM25 fused with vector scores so exact terms, IDs, and acronyms that dense embeddings miss still retrieve, the hybrid extension to pure-vector first-stage retrieval.
GitHub Copilot enterprise — retrieval over private repositories scoped to a user's access, the same private-corpus-with-permissions shape as the 10M-document deployment.