01. Recommendation / Ranking System — High-Level Design¶

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the two-stage retrieve-then-rank narrative into boxes and the three rails — serving, feedback, learning — into concrete flows. Read the architecture top to bottom, follow a homepage request and a feedback event through it, then look at what fails when the 100 ms budget is under pressure.

Architecture¶

                          ┌──────────────┐
          client ───────▶ │  Edge / API  │  (auth, request_id mint, response cache)
                          │   gateway    │
                          └──────┬───────┘
                                 │  GET /v1/homepage
                                 ▼
                       ┌───────────────────┐
                       │  Recommendation   │   orchestrates the 100 ms budget
                       │  Orchestrator     │   (fan-out, blend, business rules)
                       └─┬───────┬───────┬─┘
             retrieve    │       │       │  score
            ┌────────────┘       │       └───────────────┐
            ▼                    ▼                        ▼
   ┌─────────────────┐   ┌───────────────┐      ┌──────────────────┐
   │  Candidate Gen  │   │  Online        │      │  Ranking Service │
   │  - ANN index    │◀──┤  Feature Store │─────▶│  (batched model  │
   │  - CF neighbors │   │  (user/item/   │      │   inference)     │
   │  - trending     │   │   context)     │      └──────────────────┘
   │  - realtime src │   └───────┬────────┘
   └─────────────────┘           │ writes (streaming features)
                                 │
   ══════════════════════════════╪═══════════════  FEEDBACK RAIL (async)
                                 ▲
        client feedback ──▶ ┌──────────┐    ┌───────────────┐
                            │ Event Bus│───▶│ Stream proc    │──▶ online features
                            │ (Kafka)  │    │ (Flink)        │
                            └────┬─────┘    └───────────────┘
                                 │ (durable log, replayable)
   ══════════════════════════════╪═══════════════  LEARNING RAIL (offline)
                                 ▼
                    ┌──────────────────────┐   ┌──────────────────┐
                    │  Log / warehouse     │──▶│ Batch: training, │
                    │  (impressions+events)│   │ embeddings, ANN  │──▶ model registry
                    └──────────────────────┘   │ index, CF, pools │    + feature backfill
                                               └──────────────────┘

Read it in three bands. The top band is the serving rail: a request enters the gateway, the orchestrator fans out to candidate generation and the online feature store, then calls ranking, blends the result, and returns rows — all inside 100 ms. The middle band is the feedback rail: every impression and interaction lands on a durable event bus and is consumed by a stream processor that materializes fresh features back into the online store within seconds. The bottom band is the learning rail: the same events are archived to a warehouse where batch jobs retrain the model, recompute embeddings, rebuild the ANN index, and refresh candidate pools, publishing new artifacts to a registry that the serving tier picks up. The three rails are deliberately decoupled: the feedback firehose can spike to 350k events/second or the batch training can fail entirely, and homepage serving keeps working.

Components¶

Edge / API gateway. Terminates TLS, authenticates the user, and mints the request_id that ties an impression back to the exact model version and scores that produced it. It can hold a very short response cache (a few seconds) so a user hammering refresh does not re-run the full pipeline — but the TTL is deliberately short because the point of the system is freshness.

Recommendation orchestrator. The conductor of the 100 ms budget. It fans out to candidate sources in parallel, fetches features for the merged candidate set in one batched call, invokes the ranking service, then applies blending, deduplication, diversity, business rules, and exploration before shaping the final rows. Crucially it enforces per-stage deadlines: if candidate generation or feature fetch blows its slice, the orchestrator proceeds with whatever came back rather than missing the whole-page deadline. It is stateless and scales horizontally.

Candidate generation. Several recall-oriented sources that each nominate items, run in parallel and merged. An ANN index over item embeddings returns items near the user's embedding; collaborative-filtering neighbor lists (precomputed offline) return "users like you also watched"; a trending/popularity pool covers cold-start and gives everyone a floor; and real-time sources inject items derived from the current session (the just-finished title's similar-items list). Together they hand the orchestrator ~500 candidates. This stage is tuned for recall and speed, not precision — being wrong here just means a slightly worse candidate that ranking will downrank anyway.

Online feature store. A low-latency key-value store holding features keyed by entity: user:U, item:I, and cross features. It answers a batched multi-get for the ~500 candidates plus the user in one round trip. It is written from two directions — nightly batch backfill for slow features and the streaming pipeline for fresh ones — and read on every request. This is the component where freshness and latency collide, so its read path is the tightest.

Ranking service. Loads the current ranking model from the registry and scores the candidate set in a single batched forward pass (500 items × features → 500 scores). It runs on inference-optimized hardware, batches across candidates within a request, and exposes the model version so the feedback rail can attribute training labels to the exact model that served them. Swapping models is a registry pointer change plus a warm-up, enabling shadow and canary deploys.

Event bus (Kafka). The durable, replayable spine of the feedback rail. The client posts batched feedback; the bus absorbs the 350k-events/second peak and decouples producers (clients) from consumers (stream processor, warehouse loader). Durability matters: these events are the only record of what was shown, and losing them corrupts training.

Stream processor (Flink). Consumes the event stream and maintains near-real-time aggregates — "titles this user played in the last hour," "trending in the last 10 minutes" — writing them back into the online feature store within seconds. This is the mechanism that lets the homepage reflect recent activity without any offline job running.

Warehouse + batch/learning rail. The event archive plus the periodic jobs: model training, embedding recomputation, ANN index rebuild, CF neighbor recomputation, and candidate-pool refresh. Output artifacts land in a model/feature registry that serving components pull. This rail runs on the order of hours to a day and is entirely off the critical path.

Primary read path (render a homepage)¶

GET /v1/homepage hits the gateway, which authenticates, mints a request_id, and (on a cache miss) forwards to the orchestrator.
The orchestrator fans out to candidate sources in parallel with a ~15 ms deadline: ANN lookup on the user embedding, CF neighbor list, trending pool, and the real-time session source. It merges and dedups their output into ~500 candidates. Any source that misses its deadline is simply dropped from the merge.
It issues one batched feature fetch to the online store for the user plus all ~500 candidates, ~20 ms. Features already carry the streaming-updated recency signals, so "watched in the last hour" is present without any extra work here.
It calls the ranking service with the candidates, user, and context; the model returns 500 scores in ~40 ms as a single batched inference.
The orchestrator blends and shapes: it groups scored items into rows by theme, applies diversity (no single genre dominating), dedup across rows, business rules, and swaps ~5–10% of slots for exploration picks, ~15 ms.
It returns the ordered rows with per-item why reasons and the model_version. The gateway relays them, and the client fires impression events (rail two) as rows scroll into view.

The whole path is a parallel fan-out, one batched feature read, one batched inference, and a shaping pass — engineered so the sum of the slowest branch in each stage stays under 100 ms, not the sum of every call.

Primary write path (feedback + learning)¶

The client batches impressions and interactions and posts them to POST /v1/feedback, tagged with the originating request_id. This is fire-and-forget from the client's view; a 202 returns immediately.
The gateway drops the events onto the event bus. Nothing here is synchronous with any homepage render.
The stream processor consumes the events and updates near-real-time features (recent plays, session context, short-window trending), writing them into the online feature store within seconds — this is what makes the next homepage load reflect the activity.
In parallel, a warehouse loader archives every event durably. Impressions and interactions together form the training labels: shown-and-clicked is positive, shown-and-ignored is negative, and exploration events carry the uncertainty the model needs.
Batch jobs periodically retrain the ranking model, recompute user/item embeddings, rebuild the ANN index, and refresh CF neighbor lists and candidate pools, publishing to the registry.
Serving components pick up new artifacts via canary/shadow rollout; a bad model is caught before full traffic.

Storage choices¶

Item embeddings → in-memory ANN index (FAISS/ScaNN-style). 10M items × 256-dim int8 ≈ 2.5 GB fits in RAM on a few replicas; approximate nearest-neighbor gives sub-10 ms recall over the whole catalog, which no relational query can match. Rebuilt in batch, served read-only.
User/item/context features → sharded online feature store (Redis / Cassandra / DynamoDB-class). Chosen for single-digit-millisecond batched multi-get on ~500 keys. Sharded by entity key; ~200 GB of user features + ~10 GB item features. Dual-written by batch backfill (slow features) and the stream processor (fresh features).
Precomputed candidate pools & CF neighbor lists → key-value by user. user_id → [item_id…], refreshed daily/hourly in batch, read as one lookup at request time. Storing 200M × ~500 ids costs ~800 GB if fully materialized, so in practice you store the cheap sources (CF neighbors) and generate the expensive ones (ANN) online.
Feedback events → durable log then columnar warehouse. Kafka for the replayable stream; a columnar/lakehouse store (BigQuery, ClickHouse, S3+Parquet) for the ~1 TB/day archive queried by aggregation for training and analytics — never by point key on the hot path.
Model & feature artifacts → registry + object store. Versioned, immutable, promoted by pointer swap so rollback is instant.

Scaling¶

Read path. Everything fans out and scales horizontally: add orchestrator replicas to raise request throughput, add ANN and ranking replicas to raise per-request capacity. Sharding the online feature store by entity key spreads the batched multi-get evenly — a request for 500 candidates hits ~500 keys distributed across shards, so no shard is hot from key skew. Growing from 5,800 req/s average to the 20,000/s peak is met by autoscaling replicas, not by re-architecting; the batched-inference tier is the piece to scale first, since scoring is the 40 ms slice.

Scoring cost. At peak, 20,000 req/s × 500 candidates = 10M scores/second. With batched inference where one accelerator handles a few thousand 500-wide requests per second, this is on the order of tens of inference nodes plus replication headroom. The lever that keeps this bounded is candidate count: doubling candidates from 500 to 1,000 roughly doubles both scoring latency and the inference fleet, so recall is capped where marginal ranking quality stops paying for marginal cost.

Feedback path. The event bus is partitioned (by user or item) to absorb 350k events/second and replays on consumer lag; the stream processor scales by partition count. Because it is fully async, a 3× feedback spike raises stream-processing lag (features get a bit staler) but never touches homepage latency.

Precompute vs online, in numbers. Shifting work offline trades freshness for cost: full nightly precompute of 200M homepages is 100B scores/night ≈ 4.6M scores/s on cheap batch hardware but 24 h stale; the hybrid keeps retrieval offline and pays only the 10M scores/s online to stay fresh. The dial between them is which features and candidates are allowed to be batch-stale versus streamed-fresh.

Operational signals¶

The healthy signal is p99 whole-page latency sitting comfortably under 100 ms with the per-stage budget intact — retrieval, feature fetch, and scoring each within their slice. The first metric to degrade under trouble is usually p99 online-feature-store latency, because it is on the critical path and sensitive to a hot shard or a GC pause; when it slips, the orchestrator starts hitting its feature-stage deadline and serving requests with partial features, which shows up as a quiet drop in ranking quality before it ever shows up as a latency alert. The misleading metric is offline model accuracy (AUC/NDCG on a holdout): it can look great while live engagement sags, because the offline set does not contain the exploration and position-bias effects the live system suffers — trust the online A/B engagement metric over the offline score. The graph an experienced operator opens first during an incident is candidate-source fill rate and per-stage deadline-miss rate: if the ANN source is missing its 15 ms deadline and getting dropped, the page silently degrades toward trending-only recommendations while every top-line latency number still looks fine.

Failure modes and resilience¶

Ranking service slow or down. The orchestrator hits its scoring deadline and falls back to ordering candidates by a cheap heuristic (recent popularity, CF score) rather than the model. The page still renders, personalization degrades gracefully. Mitigate with replicas, canary deploys, and a circuit breaker around the ranker.
Online feature store hot shard. Batched multi-gets slow, feature-stage deadline misses, ranking scores on partial features. Mitigate by sharding evenly on entity key, replicating hot shards, and having the model tolerate missing features (impute rather than error).
Recency not reflected — the threaded scenario failing. A user finishes an episode and the next load does not downrank it. This is almost never a ranking bug; it is a feedback-rail lag problem: the stream processor is behind, so the "watched in the last hour" feature has not been written yet. Recognize it as freshness, not quality: watch stream-processor consumer lag, not the model. Mitigate by keeping the recency-critical features on a low-lag partition, and by using the real-time candidate source and request-time features so the just-finished title is handled inline even if the streaming aggregate lags.
Feedback firehose spike (350k/s → 1M/s). Kafka absorbs it as consumer lag; features get staler by seconds to minutes; serving is untouched. Events replay on catch-up, so no training data is lost. The failure is bounded to freshness, by design.
Bad model deployed. A retrain that regresses engagement. Mitigate with shadow scoring (run new model in parallel, compare) and canary (1% traffic) before promotion, plus instant rollback via registry pointer swap.
Feedback loop / popularity collapse. Without impression logging and exploration, the model concentrates traffic on a shrinking head and the catalog rots. Mitigate structurally: log impressions as negatives, reserve exploration slots, and monitor catalog coverage (fraction of items ever shown) as a first-class metric.
Cold-start user/item. No history to retrieve or score. Fall back to the trending/popularity pool and context-only features for users; use content features (metadata embeddings) for brand-new items until interaction data accrues.

Where this shows up in production¶

Netflix — splits the homepage into a two-stage retrieve-then-rank per row and runs continuous A/B tests, treating the page as a ranking of rows, not just items within rows.
Amazon — pioneered item-to-item collaborative filtering, the precomputed neighbor-list candidate source that gives "customers who bought this also bought" without scoring the whole catalog online.
YouTube — the canonical two-tower deep-retrieval-then-ranking architecture, where a candidate network narrows millions of videos to hundreds and a heavier ranking network orders them under a latency budget.
Meta / Instagram — runs a multi-stage ranking funnel (retrieval → lightweight ranker → heavy ranker) precisely to keep the expensive model's input set small, the same set-shrinking logic as here.
Spotify — blends multiple candidate sources (collaborative, content-based, editorial) and leans on exploration to surface long-tail tracks, the exploration-vs-exploitation dial made product.
TikTok — the extreme freshness case: near-real-time feature updates and rapid feedback incorporation so the feed reacts within a handful of interactions, the streaming-features rail pushed to its limit.
Feast / Tecton (feature stores) — productize exactly the batch-plus-streaming dual-write online store described here, materializing offline and streaming features to a low-latency serving layer.
FAISS / ScaNN (Google) — the in-memory approximate-nearest-neighbor libraries that make candidate generation over 10M embeddings a sub-10 ms lookup instead of a full scan.
LinkedIn — logs impressions with position to correct for presentation bias when training, the "log what was shown, not just what was clicked" discipline that keeps the feedback loop honest.