03. Recommendation / Ranking System — Interview Q&A

~15 min read · Part 4 of 4 (Overview → HLD → LLD → Q&A)

These are the questions an interviewer reaches for once the two-stage diagram is on the board. Each answer is the strong version, followed by the wrong answer that quietly sinks candidates.

Q1. Why two stages — candidate generation then ranking — instead of one model that scores everything? Because scoring the whole catalog per request is infeasible: 10M items × 20,000 requests/second = 2 × 10¹¹ scores/second. Candidate generation narrows 10M to ~500 cheaply (approximate-nearest-neighbor over embeddings, precomputed neighbor lists, trending pools), and only then does the heavy ranking model score those 500 — 20,000 × 500 = 10M scores/second, four orders of magnitude smaller. Retrieval is recall-oriented and fast; ranking is precision-oriented and heavy. The whole architecture exists to shrink the set the expensive model ever sees. Common wrong answer to avoid: "Run the ranking model over the full catalog and take the top N." That is 200 billion scores/second at peak — it does not matter how good the model is if it cannot run in the budget.

Q2. The homepage must render in under 100 ms. How do you actually spend that budget? Split it per stage and enforce a deadline on each: candidate retrieval ~15 ms, one batched feature fetch ~20 ms, batched scoring of 500 candidates ~40 ms, blending/diversity/business-rules/exploration ~15 ms, leaving ~10 ms for network. The budget is spent on the slowest branch per stage because candidate sources fan out in parallel and features and scores are fetched in single batched calls, not the sum of every individual call. Scoring is the biggest slice and grows linearly with candidate count, which is why you cap candidates near 500 — doubling to 1,000 blows the 40 ms slice and the whole page deadline. Common wrong answer to avoid: "Optimize the model to be faster." Model micro-optimization is a rounding error next to fanning out retrieval, batching the feature read, and capping candidate count.

Q3. What does "reflect recent activity" actually require? A user finishes an episode and reloads — trace it. This is a freshness-of-features problem, not a ranking problem. When the user finishes the episode, a play event hits the event bus and the stream processor updates that user's recent_items and short-window features in the online store within seconds. The next homepage load reads those as ordinary features, and a real-time candidate source pulls the just-finished title's neighbors so crime-drama candidates seed the next page immediately. The finished title itself scores low (a "completed today" feature suppresses it) while its neighbors score high. Nothing is re-ranked across the catalog and no batch job runs — you refresh recommendations by refreshing features and injecting real-time candidates. Common wrong answer to avoid: "Retrain or re-score the whole catalog when the user acts." Retraining takes hours and re-scoring 10M items misses the 100 ms budget by four orders of magnitude; recency lives in the feature and candidate layers.

Q4. Precompute the homepages nightly or rank in real time? Neither extreme. Full nightly precompute is 200M users × 500 candidates = 100B scores/night ≈ 4.6M scores/second on cheap batch hardware with no tail-latency pressure — but the result is up to 24 hours stale and cannot reflect recent activity. Pure real-time is always fresh but costs 10M scores/second on latency-critical hardware. The resolution is hybrid: precompute the expensive, slow-changing part offline (embeddings, ANN index, CF neighbor pools) and rank online with fresh features. Retrieval, which scans 10M items, is amortized in batch; ranking 500 candidates is cheap enough to do live and stay current. Common wrong answer to avoid: "Precompute everything nightly and cache it — it's fast and cheap." It is, and it violates the one requirement that makes this a hard problem: a homepage that ignores what you did five minutes ago.

Q5. Freshness costs money. How do you decide which features are real-time versus batch? By how much a feature moves the ranking against what it costs to keep fresh. Long-term taste changes slowly, so it stays in cheap daily batch. Session and recency signals ("watched in the last hour") change the ranking sharply, so they go through the streaming pipeline that chews ~350,000 events/second and writes into the online store within seconds. Making everything real-time would mean 350k feature writes/second plus paying that read cost on every one of 20k requests/second, for features that barely move — so freshness is bought selectively on the few features that matter, and the just-watched item is injected as a real-time candidate rather than as a recomputed feature. Common wrong answer to avoid: "Make all features real-time so it's always fresh." You pay a streaming-pipeline and online-store cost across the board for slow-moving features that do not change the answer.

Q6. Why exploration? Isn't showing the highest-scored items always best? Ranking purely by score maximizes today's clicks but starves the system of data about everything it did not show. The model then learns that only-what-it-showed is good — traffic concentrates on a shrinking head, the long tail never gets impressions, and the catalog rots. So you reserve ~5–10% of slots for higher-uncertainty items (epsilon-greedy or Thompson sampling), trading a small, measurable dip in immediate engagement for the exploration data that keeps the model honest and the catalog discoverable. The paired requirement is logging impressions, not just clicks, so shown-and-ignored becomes a training negative. Common wrong answer to avoid: "Always exploit — show the best predictions." That optimizes the current quarter and quietly destroys the model's training distribution and the catalog's coverage.

Q7. What is the feedback loop, and how does it corrupt itself if you're careless? The system trains on data it itself generated: it shows items, logs outcomes, and retrains. Two biases creep in. Presentation/position bias — items ranked #1 get clicked because they are #1, not because they are best — so you must log position and correct for it, or the model just learns to reward whatever it already puts on top. Popularity bias — without exploration and without logging impressions as negatives, the model conflates "not shown" with "not liked" and collapses onto the head. The fixes are structural: log impressions with position and a model_version, tag exploration slots, and monitor catalog coverage as a first-class metric. Common wrong answer to avoid: "Train on clicks." Clicks alone, without impressions and position, teach the model that its own past ranking was correct — a self-reinforcing loop that degrades over time.

Q8. The feedback firehose is ~350k events/second at peak. How do you handle it without hurting the 100 ms render? Keep it entirely off the serving path. The client batches impressions and interactions and posts them to a durable, partitioned event bus (Kafka) that absorbs the peak as consumer lag; a stream processor consumes it to update online features, and a warehouse loader archives ~1 TB/day for training. Because it is fully asynchronous, a 3× feedback spike raises stream-processing lag — features get staler by seconds — but never touches homepage latency, and events replay on catch-up so no training data is lost. Common wrong answer to avoid: "Update the model or the counts synchronously on each interaction." A synchronous write on the interaction path couples the firehose to render latency and turns a feedback spike into a serving outage.

Q9. A user finished an episode but the next load doesn't downrank it. What broke? Almost certainly not the model — it is feedback-rail lag. The stream processor is behind, so the recent_items / "completed today" feature has not been written before the next request reads it. Diagnose by watching stream-processor consumer lag, not offline model accuracy. Mitigate by putting recency-critical features on a low-lag partition and by leaning on the real-time candidate source and request-time features, which handle the just-finished title inline even when the streaming aggregate is a beat behind. Recognize it as a freshness problem, not a quality problem. Common wrong answer to avoid: "The ranking model needs retraining." Retraining takes hours and does nothing for a seconds-scale staleness in one feature; you would be fixing the wrong layer.

Q10. Offline the new model's NDCG is up, but you deployed it and engagement dropped. Why, and how do you catch it? Offline metrics are computed on logged data that carries the old model's exploration and position bias, so a model that scores well offline can behave worse live — the offline set does not contain the counterfactual of what the new model would have shown. Catch it before full traffic with shadow scoring (run the new model in parallel, compare distributions) and a canary at 1% of live traffic measured on real engagement, with instant rollback via a registry pointer swap. Trust the online A/B engagement metric over the offline score. Common wrong answer to avoid: "Offline metrics went up, ship it to 100%." Offline AUC/NDCG is a screening filter, not a launch decision; the live feedback loop is the only real judge.

Q11. New user, no history — what do you show, and how is it different for a new item? For a cold user, retrieval falls back to the trending/popularity pool and context-only features (device, time, locale), so the page renders something reasonable while interaction data accrues; exploration then learns their taste quickly. For a cold item with no interaction stats, use content features — a metadata embedding from genre/cast/description — so it can be retrieved and scored on similarity before it has any plays, and let exploration give it the impressions it needs to earn behavioral signal. Common wrong answer to avoid: "Return an empty or generic page for new users." A blank or identical page for everyone wastes the session and gives the system no signal; a popularity floor plus exploration is both a better experience and a better learner.

Q12. Where do you shard, and why doesn't a hot item become a hot shard? The online feature store shards by entity key (user:U, item:I), so a single request for 500 candidates spreads across ~500 keys on many shards — read load is naturally fanned out rather than concentrated. A globally popular item is one hot key, handled by replicating that key and by caching, not by re-sharding. The ANN index is held read-only in memory and replicated for throughput. Traffic skew (a trending title everyone loads) is a caching/replication problem; data skew across shards is prevented by the even entity-key distribution. Common wrong answer to avoid: "Shard by user region" or "shard by genre." Both create skew — power regions and blockbuster genres become hot shards — and neither matches the per-request access pattern of a batched multi-get across many items.

Deeper follow-ups

• How would you rank the rows themselves (which shelf goes on top), not just items within a row — and what signal tells you a user prefers "Continue Watching" over "Trending" today?
• How do you correct position bias quantitatively — inverse-propensity weighting, a position feature, or randomized-position exploration — and what are the tradeoffs?
• How would you serve recommendations with globally low latency for 200M users across regions, and what staleness would you accept in the online feature store's cross-region replication?
• How do you A/B test a ranking change without the two variants' feedback contaminating each other's training data?
• How would you detect and unwind a runaway feedback loop that has already collapsed traffic onto a small head of items?
• What changes if the business adds a hard constraint — a contractual "this title must appear in the top 3 rows for all users" — without wrecking personalization or the exploration budget?

How this round is scored

Interviewers use the recommender to see whether you reach for retrieve-then-rank before you reach for a bigger model. The strong signal is recognizing on the first pass that scoring the catalog is infeasible (2 × 10¹¹/second) and that candidate generation is the point of the whole design, then spending the 100 ms budget explicitly per stage rather than hand-waving "it'll be fast." Seniority shows up in the three tradeoff discussions — precompute vs real-time, freshness vs cost, exploration vs exploitation — where a strong candidate names both sides with numbers (24 h stale vs 10M scores/second; 8% exploration slots for long-term coverage) and resolves them rather than picking one and moving on. The candidates who have operated these systems are the ones who, when the homepage stops reflecting recent activity, look at stream-processor lag instead of the model, and who insist on logging impressions with position because they have watched a feedback loop eat a catalog. Doing the arithmetic out loud and using it to justify the candidate cap, the feature-freshness split, and the exploration budget is what separates "correct" from "senior."