00. Design a Recommendation / Ranking System¶

~20 min read · Level: advanced · Part 1 of 4 (Overview → HLD → LLD → Q&A)

Problem¶

A recommendation system decides what to show a user and in what order. It is the machinery behind the Netflix homepage, the Amazon "recommended for you" shelves, the YouTube watch-next rail, and the Spotify weekly mix. The user does not type a query; the system infers intent from who they are and what they have done, then assembles a ranked layout out of a catalog far too large to browse. The visible artifact is a page of rows — "Continue Watching," "Because You Liked X," "Trending Now" — but each row is a ranking problem, and the page as a whole is a packing problem on top of many rankings.

What makes this a hard system-design question is not the math of any single model. It is that the ranking has to happen per user, per request, under a hard latency budget, over a catalog of millions, and it has to feel current. You cannot score ten million items for two hundred million people on every page load, and you cannot serve last night's answer to someone who just finished a movie thirty seconds ago. The architecture exists to resolve that squeeze.

Thread one scenario through the whole design: build a personalized homepage for 200,000,000 users where the ranked rows must render in under 100 ms and reflect recent activity. Concretely, a user finishes an episode of a crime drama, returns to the home screen, and expects the very next load to surface more crime drama and downrank what they just finished — all while the page paints in under a tenth of a second. That single sentence — 200M users, sub-100 ms, reflect recent activity — pulls in opposite directions at every layer, and every decision below is an answer to it.

Functional requirements¶

Personalized homepage: given a user and a context (device, time, entry point), return an ordered set of rows, each row an ordered list of items with a reason ("Because you watched…").
Candidate generation: narrow the full catalog (millions of items) down to a few hundred plausible items for this user, from several sources (behavioral neighbors, embedding similarity, trending, continue-watching).
Ranking: score those candidates with a model that combines user, item, and context features, and order them.
Recency reflection: incorporate the user's activity from the last few minutes, not just last night's batch.
Feedback capture: log impressions and interactions (what was shown, what was clicked/played/skipped) to feed training and online features.
Exploration: reserve some slots for items the model is uncertain about, so the catalog keeps getting discovered and the training data does not collapse onto what was already popular.

De-scoped for this round, and worth naming so the interviewer hears it as a choice: the offline model training pipeline (feature engineering, labeling, hyperparameter search) and the search/query path. We assume trained models exist and are deployed; we design the system that serves recommendations and closes the feedback loop, not the ML research workflow. We also set aside cross-device identity resolution and the editorial/business-rule layer (contractual "must-promote" titles) beyond a hook for it.

Non-functional requirements¶

The single dominant constraint is read-path tail latency under a fixed per-request compute budget. Everything else bends to it.

Latency: the ranked rows must render in under 100 ms at p99, not just at the median. Personalization is worthless if it makes the page feel slow — a slow homepage loses more engagement than a slightly-worse-ranked fast one.
Freshness: the page must reflect activity from the last few minutes. This is in direct tension with latency and cost, and resolving that tension is the core of the design.
Availability: the homepage must always render something. A recommender that returns nothing is worse than one that returns a decent non-personalized fallback. Target four nines on the read path, with graceful degradation to popularity-based rows.
Scalability: 200M users, tens of thousands of homepage requests per second at peak, and a feedback firehose in the hundreds of thousands of events per second.
Quality under feedback: the system trains on data it itself generated, so it must log impressions (not just clicks) and inject exploration, or it will quietly poison its own training set.

Scale estimation¶

Take 200M total users, of whom roughly 100M are daily active. If each active user loads the homepage about five times a day, that is 100M × 5 = 500M homepage requests/day, or 500M / 86,400 ≈ 5,800 requests/second on average. Apply a peak factor of ~3.5× for prime-time concentration and design the read path for ~20,000 homepage requests/second at peak. This is the number the whole serving tier is sized around.

Each request cannot touch the whole catalog. With ~10M items, scoring all of them per request would be 20,000 × 10M = 2 × 10¹¹ score computations per second — absurd. So retrieval narrows the catalog to ~500 candidates per user per request, and ranking scores only those: 20,000 × 500 = 10,000,000 item-scores/second at peak. Ten million scores per second is large but tractable on a fleet of batched-inference nodes; two hundred billion is not. That reduction from 10M to 500 is not an optimization — it is the reason the system can exist.

The 100 ms p99 budget has to be spent, not assumed. A workable split: candidate retrieval ~15 ms, online feature fetch ~20 ms, model scoring of 500 candidates ~40 ms, blending/business-rules/dedup ~15 ms, leaving ~10 ms for network and serialization. Notice scoring is the biggest single slice and grows linearly with candidate count — which is exactly why you keep candidates near 500, not 5,000.

The feedback firehose dwarfs the request rate. Impressions dominate: a scrolling user generates many "this item was shown" events plus clicks, plays, and skips. Estimate ~100 events per DAU per day → 100M × 100 = 10^10 = 10 billion events/day, or ~115,000 events/second average and ~350,000/second at peak. This stream is the raw material for both online features and model training, and at 100 bytes/event it is ~10^12 bytes/day ≈ 1 TB/day of raw feedback before compression.

Storage reconciles cleanly. Item embeddings for the ANN index: 10M × 256-dim, quantized to int8, ≈ 2.5 GB — small enough to hold fully in memory on a few nodes. User embeddings: 200M × 256 × 1 byte ≈ 51 GB, sharded. The online feature store, at ~1 KB of features per user, is 200M × 1 KB ≈ 200 GB plus ~10 GB for item features — a sharded in-memory or SSD-backed key-value cluster. None of this is a big-data storage problem; the pressure is all on latency and freshness, not bytes.

API sketch¶

GET /v1/homepage?user_id=U&device=tv&context={time,entry}
  200: { rows: [ { row_id, title: "Because you watched …",
                   reason, items: [ {item_id, score, why} … ] } … ],
         model_version, request_id }
  # request_id is echoed on every impression so training can join shown↔score

POST /v1/feedback                      # async, batched by the client
  body: { user_id, request_id,
          events: [ {item_id, type: impression|click|play|skip|thumb,
                     position, row_id, ts} … ] }
  202: accepted

# --- internal contracts ---
GET  /v1/candidates?user_id=U&context=…     -> { candidates: [item_id…], sources: {…} }
MGET /v1/features {entities:[user:U, item:…]} -> { feature_vectors }   # online store
POST /v1/rank { user_id, candidates, context } -> { ordered: [{item_id, score}…] }

Solutioning¶

Start from the impossible number and the design falls out. Scoring 10M items for 200M users on demand is 2 × 10¹¹ ops/second; the only escape is to not rank the catalog. So the system is two stages: candidate generation (retrieval) narrows 10M items to ~500 cheaply, and ranking scores those 500 expensively. This is the load-bearing reframing: a recommendation system is not a scoring problem, it is a candidate-generation problem — the whole architecture exists to shrink the set the expensive model ever sees. Retrieval is recall-oriented and fast (approximate nearest-neighbor over embeddings, precomputed neighbor lists, trending pools); ranking is precision-oriented and heavy (a model over hundreds of features). Getting the split right is most of the design.

The first defining tradeoff is precompute versus real-time. You could compute every user's ranked homepage nightly in batch: 200M users × 500 candidates = 100 billion scores spread over a ~6-hour window is ~4.6M scores/second sustained — cheaper per score on batch hardware, no tail-latency pressure, trivially cached. But the result is up to 24 hours stale, which flatly violates "reflect recent activity." Pure real-time ranking on every request is always fresh but costs 10M scores/second at peak on latency-critical hardware. The resolution is neither extreme: precompute the expensive, slow-changing part (candidate pools and embeddings) offline, and rank online with fresh features. Retrieval — the part that scans 10M items — is amortized in a nightly/hourly batch; ranking 500 candidates is cheap enough to do live. This is the hybrid every large recommender converges on, and the memory hook is: you don't refresh recommendations by re-ranking the catalog; you refresh them by refreshing features and injecting real-time candidates.

The second tradeoff is freshness versus cost, and it is where "reflect recent activity" gets paid for. Features come in three temperatures. Batch features (long-term taste, computed daily) are cheap and stale. Streaming features (last-hour aggregates, updated by a Flink/Kafka pipeline within seconds) cost a stream processor chewing through 350k events/second. Request-time features (the item they finished 30 seconds ago) are freshest and cheapest per user but must be assembled inline. Making every feature real-time would mean writing 350k feature updates/second to the online store and paying that read cost on every one of 20k requests/second; instead you keep long-term taste in cheap daily batch, push only a small set of session/recency features through the streaming path, and inject the just-watched item as a real-time candidate source so the homepage visibly reacts without recomputing anything. Freshness is bought selectively, on the few features that move the ranking, not uniformly.

The third tradeoff is exploration versus exploitation, and it is the one juniors skip. Ranking strictly by predicted score always shows the model's safest bets, which maximizes today's clicks but starves the system of data about everything it did not show — the model then learns that only-what-it-showed is good, a feedback loop that concentrates traffic on a shrinking head and lets the catalog rot. So you deliberately spend a slice of slots — say 5–10% — on higher-uncertainty items (epsilon-greedy or Thompson-sampling bandits), trading a small measurable dip in immediate engagement for the exploration data that keeps the model honest and the catalog alive. The corollary is that you must log impressions, not just clicks: the training set has to know what was shown-and-ignored, or the model conflates "not shown" with "not liked."

The result is a system with three rails. A read/serving rail does retrieve-then-rank under the 100 ms budget with heavy caching and graceful fallback. A feedback rail absorbs the 350k-events/second firehose asynchronously and never touches the serving path. And a learning rail turns that logged feedback into fresh streaming features (minutes) and periodically retrained models (hours to days). The next file lays these out as components; the one after pins down the schemas, the retrieve-then-rank algorithm, and the concurrency corners where a recommender actually breaks.