02. RAG Knowledge Assistant — Low-Level Design¶
~20 min read · Part 3 of 4 (Overview → HLD → LLD → Q&A)
The HLD named the boxes. This file opens the three that carry the design's weight — the ingestion pipeline (chunk, embed, upsert with versioning), the two-stage retriever (ANN plus rerank with ACL), and the grounded generator — and pins down the data, the algorithms, and the concurrency corners where a RAG system actually breaks.
Data models¶
The vector index entry is deliberately thin. Everything bulky lives elsewhere; the index holds only what the ANN search and its filters need in RAM.
VectorEntry (in the ANN index, held in memory)
chunk_id uint64 // stable, deterministic: hash(doc_id, ordinal)
vector int8[1024] // scalar-quantized embedding, ~1 KB
doc_id uint64 // for fetch + per-document delete
acl_tag uint64 // permission group id(s) for the ACL pre-filter
index_version uint16 // embedding-model version guard
Two non-obvious choices. The chunk_id is deterministic — derived from (doc_id, ordinal) — so re-embedding a document produces the same ids for surviving chunks and lets us compute exactly which old chunks are now orphans. And the acl_tag lives on the vector itself so the ANN search can filter candidates during traversal, rather than fetching everything and filtering after (a post-filter can return fewer than k results and is a leak waiting to happen).
The chunk text sits in a separate KV store, keyed by the same id:
ChunkRecord (KV store, key = chunk_id)
chunk_id uint64
doc_id uint64
ordinal int // position within the document
text string // context-prefixed 512-token chunk
char_start int // offsets into the source doc, for citation
char_end int
token_count int
index_version uint16
The char_start/char_end offsets are what let a citation deep-link to the exact passage in the source, not just the document. The document record is the transactional system of record:
CREATE TABLE document (
doc_id BIGINT PRIMARY KEY,
source VARCHAR(32) NOT NULL, -- 'confluence','jira','gdrive'
external_id VARCHAR NOT NULL,
version BIGINT NOT NULL, -- monotonic per doc, bumped on edit
content_hash CHAR(64) NOT NULL, -- skip re-embed when unchanged
acl JSONB NOT NULL, -- principals allowed to see this doc
index_version SMALLINT NOT NULL, -- model version its vectors were built with
chunk_count INT NOT NULL,
indexed_at TIMESTAMP NOT NULL, -- freshness clock
status SMALLINT NOT NULL DEFAULT 1 -- 1=active, 0=tombstoned
);
version orders concurrent edits, content_hash skips needless re-embeds, acl drives the retrieval filter, and indexed_at is the value that answers "was the 5-minute-ago edit picked up yet?" The change event that flows through the queue is small:
Component internals¶
Component 1 — Ingestion pipeline (chunk, embed, versioned upsert)¶
Responsibility: turn a change event into index entries exactly once, newest-write-wins, with no orphaned chunks left behind.
class Ingestor:
def handle(event: ChangeEvent) -> None
def _chunk(text: str) -> list[Chunk] # 512 tok, 64 overlap
def _prefix(chunk: Chunk, doc: Doc) -> str # title/section context
def _upsert(doc_id, chunks, vectors, version) -> None # atomic-enough commit
The handler is where versioning and dedupe live:
def handle(self, event):
if event.op == "delete":
return self._purge(event.doc_id, event.version) # fail-closed removal
current = document.get(event.doc_id)
if current and event.version <= current.version:
return # stale event — a newer edit already won
doc = event.content or fetch_source(event.uri)
h = sha256(doc.text)
if current and h == current.content_hash:
document.touch_metadata(event.doc_id, event.acl) # unchanged: no re-embed
return
chunks = self._chunk(doc.text)
for c in chunks:
c.text = self._prefix(c, doc)
c.chunk_id = det_id(event.doc_id, c.ordinal) # deterministic id
vectors = embedder.embed_batch([c.text for c in chunks], version=CURRENT_V)
self._upsert(event.doc_id, chunks, vectors, event.version)
_upsert writes the new chunks, then removes the orphans — the chunks the previous revision had that this one no longer produces — and commits the document row last so nothing becomes queryable until its text and vectors are both in place:
def _upsert(self, doc_id, chunks, vectors, version):
new_ids = {c.chunk_id for c in chunks}
old_ids = set(document.get(doc_id).chunk_ids) if exists else set()
chunk_store.mput(chunks) # text first
vector_index.upsert([(c.chunk_id, v, doc_id, acl_tag(doc_id), CURRENT_V)
for c, v in zip(chunks, vectors)])
orphans = old_ids - new_ids
vector_index.delete(orphans); chunk_store.delete(orphans)
document.compare_and_set(doc_id, expect_version_lt=version, # newest wins
version=version, content_hash=h,
chunk_count=len(chunks), indexed_at=now())
Component 2 — Two-stage retriever (ANN + rerank, ACL-filtered)¶
Responsibility: surface the ~8 chunks most likely to answer the question, from only the documents this user may see, in well under the latency budget.
def retrieve(self, question, user, k=8):
qvec = embedder.embed(question, version=CURRENT_V) # MUST match index version
principals = user.acl_principals()
candidates = vector_index.search( # ~30 ms over 60M vectors
qvec, top=100,
filter=AclMatch(principals), # in-search, fail-closed
index_version=CURRENT_V)
texts = chunk_store.mget([c.chunk_id for c in candidates])
scored = reranker.score(question, texts) # ~200 ms, 100 pairs
scored = mmr_diversify(scored, lambda_=0.5) # kill near-duplicates
top = scored[:k]
if not top or top[0].score < TAU: # confidence gate
return Abstain(reason="no_grounded_source")
return top
The ACL match runs inside the index traversal so a forbidden chunk is never even a candidate; the confidence gate TAU is the abstain lever; and mmr_diversify trades a little raw relevance for source diversity so the 8 slots aren't 8 copies of one paragraph.
Component 3 — Grounded generator (cite, then verify)¶
Responsibility: synthesize an answer only from the retrieved chunks, cite each claim, and refuse to emit an unverifiable citation.
def answer(question, chunks):
context = "\n\n".join(f"[{i+1}] {c.text}" for i, c in enumerate(chunks))
prompt = SYSTEM_GROUNDED + f"\nContext:\n{context}\nQuestion: {question}"
draft = llm.generate(prompt, stream=True) # 1.5–2.5 s
markers = extract_markers(draft) # the [n] references used
for n in markers:
if n < 1 or n > len(chunks):
raise UngroundedCitation(n) # model invented a source
return Answer(text=draft,
citations=[to_citation(chunks[n-1]) for n in sorted(set(markers))])
The system prompt instructs the model to answer only from the numbered context and to attach a [n] to every claim; the verifier then rejects any marker that doesn't map to a real chunk, closing the gap where a model cites a source that isn't there.
Core algorithm — the 5-minute freshness path, stepped through¶
This is the scenario made mechanical. An HR editor fixes the parental-leave policy (document 42) at t=0, and a colleague must get the corrected, cited answer by t=5 min.
- t=0s — capture. The Confluence connector sees document 42 change and emits
{doc_id:42, op:upsert, version:7, acl:[hr, all-staff], content:<new text>}onto the queue. The stored record is atversion:6. - t≈3s — pickup. An ingestion worker dequeues the event. It reads document 42's record: incoming
version 7 > stored 6, so not stale. It hashes the new text; the hash differs from the storedcontent_hash, so a re-embed is warranted (an unchanged save would have short-circuited here for free). - t≈4s — chunk. The new text is ~3,000 tokens → 6 chunks at 512/64, each prefixed with "Parental Leave Policy › Contractors". Deterministic ids
det_id(42,0..5)are assigned. The previous revision had 8 chunks (it was longer), so ordinals 6 and 7 will be orphans. - t≈4.2s — embed. The embedder turns 6 chunk texts into 6 vectors with model
vN. This is ~3,000 tokens of embedding work — a couple hundred milliseconds — versus the 30 billion tokens a full corpus rebuild would touch. That ratio,6 chunksagainst60,000,000 chunks, is the whole argument for incremental upsert. - t≈4.5s — upsert + cleanup. The worker writes the 6 chunk texts, upserts the 6 vectors, then deletes orphaned chunks
det_id(42,6)anddet_id(42,7)from both the index and the chunk store — otherwise a query could still retrieve and cite a paragraph the editor deleted. It commitsdocument.version=7, indexed_at=t. - t=5min — query. The colleague asks "how much parental leave do contractors get?". The query embeds with
vN, the ANN search (ACL-filtered to their principals) now returns the new chunkdet_id(42,2), the reranker floats it to rank 1 with a score aboveTAU, and the generator answers citing document 42, version 7, snippet drawn fromchar_start/char_endof the corrected passage.
Had the design chosen a nightly rebuild instead, step 6 would have returned version 6 — the old policy, confidently cited — until the batch ran that night. Incremental upsert is the difference between a 5-minute freshness window and a 24-hour one, bought for the cost of embedding six chunks.
Sequence diagram — a query end to end¶
Client Orchestrator Embedder VectorIndex ChunkStore Reranker LLM
│ question │ │ │ │ │ │
├─────────────▶│ │ │ │ │ │
│ ├─ embed(q,vN) ─▶│ │ │ │ │
│ │◀─── qvec ──────┤ │ │ │ │
│ ├─ search(qvec, top=100, ACL) ─▶│ │ │ │
│ │◀──── 100 chunk_ids ───────────┤ │ │ │
│ ├─ mget(ids) ──────────────────────────────▶│ │ │
│ │◀──────────── 100 texts ────────────────────┤ │ │
│ ├─ score(q, texts) ───────────────────────────────────▶ │ │
│ │◀──────────── reranked top-8 ───────────────────────────┤ │
│ │ (top score < TAU? ── yes ──▶ 204 abstain) │ │
│ ├─ generate(prompt with [1..8]) ──────────────────────────────────▶ │
│ │◀─────────────── streamed answer + [n] markers ─────────────────────┤
│ ├─ verify markers → resolve citations │ │
│◀─ answer + citations ─┤ │ │ │ │ │
One embed, one ACL-filtered search, one batched fetch, one rerank, one generation — the abstain branch cuts out before generation when retrieval confidence is too low.
Concurrency and edge cases¶
- Out-of-order edits (race): document 42 is edited twice quickly (v7 then v8), and the events arrive reordered. The
compare_and_set(expect_version_lt=version)on the document row makes newest-write-wins deterministic: v8 commits, and v7 arriving late is rejected because 7 is not greater than the stored 8. No lock, just a monotonic version guard. - Idempotent reprocessing: replaying the queue after a bug is a no-op for unchanged documents — the content-hash check short-circuits before any embedding — and deterministic chunk ids mean re-upserting produces identical entries rather than duplicates.
- Orphaned chunks: when a document shrinks from 8 chunks to 6, ordinals 6–7 must be deleted from both stores, or a query can retrieve text the editor removed and cite deleted content. The set-difference
old_ids - new_idsover deterministic ids makes this exact rather than a guess. - Embedding-model version skew: a query embedded with
vN+1against avNindex returns meaningless neighbors — the silent-corruption failure. Handling: refuse cross-version search (theindex_versionguard), and on a model upgrade build a parallelvN+1index, dual-write during the transition, and cut queries over atomically once it's complete, then retirevN. - Read-your-writes for the editor: the HR editor expects to see their own 5-minute-ago change immediately. The
indexed_at/versionon citations makes freshness visible, and the editor's own session can force a synchronous re-index of the document they just touched rather than waiting on the queue. - ACL change after indexing: if a user loses access to document 42, a stale semantic-cache entry could still surface it. Handling: the ANN filter always uses live principals (fail closed), and semantic-cache entries are invalidated whenever a cited document's ACL or version changes — permission is never served from a cache.
- Tombstone before purge: a deleted document is marked
status=0in metadata immediately (so it's excluded even before cleanup runs) and its vectors are purged from the index synchronously, because a lazily-deleted vector is a document that can still be retrieved and cited after deletion — unacceptable for confidential or legally-removed content. - Duplicate crowding: identical passages copied across many documents produce near-identical vectors that fill the top-100 with one answer's clones. Dedup by content hash at ingestion removes exact copies, and MMR at rerank spreads the final 8 across distinct sources so the citation list is genuinely diverse.