03. RAG Knowledge Assistant — Interview Q&A

~15 min read · Part 4 of 4 (Overview → HLD → LLD → Q&A)

These are the questions an interviewer actually asks once the two pipelines are on the board. Each answer is the strong version, followed by the wrong answer that quietly sinks candidates.

Q1. Why two-stage retrieval — ANN then a reranker — instead of just returning the vector search's top-k? Because the two stages optimize different things. The ANN search uses a bi-encoder: it embeds question and chunks separately and compares vectors, which is cheap enough to scan 60M chunks in ~30 ms but blurry, since it never reads the question and chunk together. So you over-fetch — top-100, not top-8 — to get high recall cheaply, then a cross-encoder reranker reads each (question, chunk) pair jointly and reorders them, which is far more accurate but too slow to run over millions. The split gives you recall from stage one and precision from stage two. And the latency budget makes it a free call: the reranker's ~200 ms is under 10% of a ~2 s generation, so you buy a large quality lift for a rounding error of time. Common wrong answer to avoid: "The vector search already returns the most relevant results, so a reranker is redundant." The bi-encoder score is a coarse proxy; the chunk that actually answers the question is often ranked 20th by ANN and 1st by the reranker.

Q2. How do you choose chunk size, and what does the wrong size cost? It's a recall-versus-fragmentation tradeoff with real numbers. At 256 tokens you get pinpoint chunks but ~120M vectors, double the index, and facts that span a boundary get split so neither half ranks. At 1,024 tokens you halve the index to ~30M vectors and keep context intact, but each chunk dilutes the one relevant sentence among a thousand tokens, dropping recall for specific facts, and fewer distinct chunks fit the context window. 512 tokens with 64-token overlap is the resolution: the overlap stops a fact being guillotined at a boundary, and the size keeps chunks topically tight. Where a chunk still loses its bearings — "the limit is 30 days" with no subject — prepend a context prefix (title and section) at ingestion so the chunk carries its own meaning. Common wrong answer to avoid: "Use the biggest chunks the context window allows, to give the model more information." Big chunks bury the relevant sentence and cut how many distinct sources you can retrieve; more tokens is not more signal.

Q3. Someone edits a policy and a colleague asks about it 5 minutes later — how does the answer reflect the edit over a 10M-document corpus? Not by reindexing. A connector captures the edit as a change event; an ingestion worker re-chunks and re-embeds only that document's ~6 chunks (~3,000 tokens, a couple hundred milliseconds), upserts those 6 vectors, deletes any orphaned chunks the shorter revision no longer has, and commits the new version — all within seconds of the save. Five minutes later the ANN search returns the new chunk, the reranker floats it to the top, and the generator cites the document at its new version. The contrast is the whole point: a nightly full rebuild touches 60M chunks and 30B tokens and wouldn't reflect the edit until tomorrow. Freshness here is an incremental-upsert problem, not a reindex problem — six chunks, not thirty billion tokens. Common wrong answer to avoid: "Rebuild the index on a schedule (say nightly) so it stays current." A nightly batch gives a 24-hour freshness window; the edited policy stays wrong all day, cited confidently.

Q4. The assistant gives a confident answer that's flat wrong. Where's the bug? Almost always in retrieval, not generation. If the passage that answers the question wasn't in the top-k the reranker passed to the model, the model had nothing correct to work from and filled the gap fluently — a wrong RAG answer is usually not a generation problem, it's a retrieval-recall problem. So the fix path is: check whether the right chunk was even retrieved (a recall miss — bad chunking, model-version skew, or an over-aggressive ACL filter), then whether it was retrieved but ranked out (a reranker problem), and only last whether it was in-context but the model ignored it (a genuine generation issue, the rarest). Control the hallucination structurally: pass only retrieved chunks, require and verify citations, and abstain when the top reranked score is below threshold rather than answering. Common wrong answer to avoid: "Improve the prompt / use a bigger model." Prompt tuning can't surface a chunk that retrieval never returned; you'd be polishing the wrong stage.

Q5. How do you make sure a user only gets answers from documents they're allowed to see? Filter by permission inside the ANN search, so a forbidden chunk is never a candidate and can't reach the model's context — the acl_tag lives on each vector and the search takes the user's principals as a filter during traversal. Critically, this fails closed: missing or ambiguous ACL data excludes the chunk rather than including it, and a tombstoned document's vectors are purged synchronously so a deleted doc can't be retrieved after removal. Security lives at retrieval, not generation, because anything that reaches the prompt can leak into the answer. Common wrong answer to avoid: "Retrieve everything, then filter the citations / ask the model not to reveal restricted docs." A post-filter can be bypassed and the model can paraphrase restricted content it was shown; once a forbidden chunk is in the context, you've already lost.

Q6. 60M vectors — how much memory does the index take, and how do you cut the cost? At 1,024 dims and float32, each vector is ~4 KB, so 60M vectors are ~240 GB raw, ~360 GB with HNSW graph overhead — expensive to keep RAM-resident. Apply int8 scalar quantization and each vector drops to ~1 KB, cutting the index to ~60 GB, which fits in memory across a few shards for ~30 ms search. Quantization costs a little recall, but that's exactly what the reranker repairs: over-fetch top-100 on the slightly-lossy quantized search, then let the cross-encoder reorder precisely. So the memory saving is nearly free because a later stage compensates for it. Common wrong answer to avoid: "Keep full float32 vectors for accuracy" or "put the index on disk to save RAM." Full precision quadruples memory for marginal recall the reranker recovers anyway; a disk-resident ANN blows the latency budget.

Q7. You need to upgrade the embedding model. What breaks, and how do you roll it out? The trap is that a query embedded with the new model against an index built with the old one returns meaningless neighbors — with no error, just quietly worse answers, because the two vector spaces aren't comparable. So every vector and every query carries an index_version tag and the search refuses to mix versions. To upgrade, build a parallel index with the new model over the whole corpus (the ~16 GPU-hour batch, parallelized), dual-write new edits to both during the transition, then atomically cut queries over to the new version and retire the old index. You cannot incrementally embed new documents with a new model into an old index. Common wrong answer to avoid: "Just swap in the new embedding model for new documents going forward." Now the index holds two incomparable vector spaces and retrieval silently degrades for every query.

Q8. Where does the latency budget go, and can you afford a heavier reranker? Trace it: query embedding ~40 ms, ANN search ~30 ms, reranking 100 candidates ~200 ms, generation 1.5–2.5 s. Generation dominates — it's 80–90% of the ~3 s end-to-end. That asymmetry is the design's lever: since retrieval and reranking together are under 300 ms against a two-second generation, you spend generously on retrieval quality — a heavy cross-encoder, an over-fetch of 100 — because it's a rounding error next to the cost you're already paying to generate. The principle isn't "make retrieval fast," it's "retrieval is cheap relative to generation, so make it good." Common wrong answer to avoid: "Drop the reranker to save latency." You'd save ~200 ms off a ~3 s request — 7% — and lose most of your answer-quality lift; you're optimizing the wrong stage.

Q9. What's your freshness strategy's failure mode, and how do you detect it? The failure is ingestion lag. If the change queue backs up — a bulk import of 100k documents, or an embedder outage — a document edited 5 minutes ago isn't indexed yet, so the assistant answers from the old revision and cites it confidently, with no error anywhere. Detect it by watching queue depth and edit-to-queryable p95 against the 5-minute SLO, and mitigate by prioritizing small interactive edits ahead of bulk backfills in the queue and by surfacing indexed_at/version in every citation so a reader can see how fresh the source is. The dangerous property is silence: stale answers look identical to fresh ones unless you measure freshness directly. Common wrong answer to avoid: "Freshness is fine because ingestion is asynchronous and eventually consistent." Eventually-consistent with an unbounded lag means a policy can be wrong for hours; you need a measured SLO on edit-to-queryable, not just "eventually."

Q10. How do you know retrieval is actually good — how do you evaluate this system? Offline, against a labeled set: hold a set of questions with known answer-documents and measure recall@k (is the right chunk in the top-k the reranker returns) and rank quality, because that's the metric that gates everything downstream. Online, track citation-resolution rate (do cited markers map to real chunks), abstain rate (are you refusing when you should), and faithfulness (does the answer's claim actually appear in its cited chunk, which an LLM-judge can score). User thumbs-up is a lagging, biased signal — users reward confident, fluent answers whether or not they're correct — so it's a sanity check, not a primary metric. The point is to measure retrieval and groundedness directly rather than inferring quality from generation health. Common wrong answer to avoid: "Watch user satisfaction / thumbs-up rate." Fluent hallucinations get thumbs-up; a confident wrong answer scores well on satisfaction and terribly on faithfulness.

Q11. When would you add keyword (BM25) search alongside vector search? When exact terms matter and dense embeddings blur them — product SKUs, error codes, ticket numbers, acronyms, rare proper nouns. Embeddings capture semantic similarity but can miss an exact-string match ("error TX-4471" retrieves semantically-near incidents, not the one document that names that code). Hybrid retrieval runs BM25 and vector search in parallel and fuses the scores (reciprocal-rank fusion), so exact matches and semantic matches both surface into the candidate set the reranker then orders. For a corpus full of identifiers and jargon — internal tickets and design docs — the hybrid recall lift is large and cheap. Common wrong answer to avoid: "Vectors are strictly better than keyword search, so BM25 is obsolete." Dense retrieval underperforms on exact-term and out-of-vocabulary queries; the two are complementary, not a replacement.

Q12. The generation model times out or is down. What does the user get? A degraded but useful response, not an error. Generation is the least reliable, highest-latency stage, so on timeout the orchestrator falls back to returning the top reranked citations directly — "here are the most relevant sources" with their snippets and links — because retrieval already found and verified the passages; only the synthesis failed. The user still gets grounded, clickable sources they can read themselves. Because retrieval and generation are separate stages, a generation outage never takes down the ability to find documents. Common wrong answer to avoid: "Return a 500 / retry until the model responds." An error wastes the retrieval you already did; the verified citations are valuable on their own and should be served.

Deeper follow-ups

• How would you answer a multi-hop question whose answer requires joining facts from two documents that no single chunk contains?
• A single source document is longer than the context window — how do you retrieve from it without losing the parts you didn't chunk into the top-k (hierarchical / summary-of-summaries retrieval)?
• How do you isolate tenants so one customer's queries can never retrieve another's chunks in a shared index — separate indexes, namespace filters, or per-tenant ACL tags, and what does each cost?
• Two retrieved sources contradict each other (an old and a new policy both indexed) — how does the system decide which to trust, and how does recency factor into ranking?
• How would you add a cheap-model fast path for easy questions and route only hard ones to the expensive generator, and how do you decide which is which?
• How do you defend against prompt injection embedded in an ingested document ("ignore previous instructions") that reaches the model as retrieved context?

How this round is scored

Interviewers use the RAG assistant to see whether you locate correctness in retrieval rather than in the model. The strong signal is treating this as a two-stage retrieval problem with grounding and abstention built in, and diagnosing a wrong answer as a recall miss before reaching for a bigger model or a better prompt. Seniority shows up in the tradeoffs carried with numbers — chunk size against index size and recall, quantization against memory, the reranker against the generation budget, incremental upsert against a nightly rebuild — where you name both sides and pick with a reason. The freshness and security thinking separates people who've shipped enterprise RAG from those who've only prototyped it: measuring edit-to-queryable latency, filtering ACL inside retrieval and failing closed, versioning the embedding model so an upgrade doesn't silently corrupt the index. Doing the back-of-envelope math — 10M docs to 60M chunks to 60 GB quantized — and using it to justify choices rather than as decoration is what pushes an answer from "correct" to "senior."