Skip to content

00. Design a Feature Store + Training Pipeline

~20 min read · Level: advanced · Part 1 of 4 (Overview → HLD → LLD → Q&A)

Problem

A feature store is the data layer that sits between raw events and machine-learning models. It computes the signals a model consumes — a card's transaction count in the last five minutes, a merchant's 90-day average ticket size, a device's age on the platform — and serves those same signals in two very different settings: to a training job that reads months of history, and to a live model that reads one entity's current values in milliseconds. This is the system behind Uber's Michelangelo, Airbnb's Zipline, Tecton, and Feast. The job it does is deceptively narrow: make sure the number a model reads in production is computed the same way, and means the same thing, as the number it learned from during training.

The difficulty is that these two settings pull in opposite directions. Training wants to scan enormous history and reconstruct, for every labeled example, exactly what was known at that instant — no peeking at the future. Serving wants a single entity's freshest values back before a timeout fires. Build these as two independent pipelines and they drift apart within a sprint; the model trained on one distribution and scores on another, and nobody notices until the metrics quietly rot.

Thread one scenario through the whole design: a payments platform's fraud model, scoring authorizations in real time. At peak the platform runs 10,000 scoring requests per second, and each request needs features for four entities — the card, the user, the merchant, and the device — so the online store sees ~40,000 feature reads per second. The fraud team's hard requirement is the one every feature store exists to satisfy: the model must read the same feature values at 10 ms serving time that it was trained on — no train/serve skew, no label leakage. Every decision below is measured against that sentence.

Functional requirements

  • Feature definitions & registry: register a feature view — its entities, the features it produces, the transformation that computes them, and its freshness/TTL — as versioned metadata.
  • Offline (training) retrieval: given a spine of (entity_ids, event_timestamp, label), return a point-in-time-correct training dataset — each row joined to feature values as they were just before that timestamp.
  • Online (serving) retrieval: given an entity set and a feature service, return the current feature vector with a p99 under 10 ms.
  • Materialization: move computed feature values from the offline store into the online store so serving reads the same values training saw.
  • Feature pipelines: run both batch transformations (daily/hourly over history) and streaming transformations (aggregates fresh within seconds).
  • Monitoring: track per-feature freshness, training/serving skew, and value drift.

De-scoped for this round, and worth naming so the interviewer knows it is a choice: model training itself (we produce the dataset, not the model), model serving/inference (we serve features to it), a full feature-engineering notebook UX, and automated feature discovery/recommendation. These sit beside the core and do not change its shape.

Non-functional requirements

The single dominant constraint is consistency between what is trained and what is served — train/serve parity plus point-in-time correctness. A plain cache in front of a warehouse would hit every latency and throughput target and still fail the only requirement that matters, because it would serve values the training path never guaranteed to match. Everything else is a supporting constraint:

  • Latency: online retrieval p99 under 10 ms for a full feature vector; this is a hard SLA because feature reads are one hop inside a larger fraud-scoring budget of ~50 ms that itself sits inside a payment-authorization timeout.
  • Freshness: streaming features (velocity, counts) must reflect events within seconds; batch features (long-window averages) may lag hours — freshness is chosen per feature, not globally.
  • Availability: the online store must stay up; a fraud model that cannot read features has to choose fail-open (approve, lose fraud coverage) or fail-closed (decline, lose revenue), and neither is free.
  • Durability & correctness: the offline store is the system of record for history; a point-in-time join that leaks even one future value produces optimistic training metrics that collapse in production.

Scale estimation

Take the fraud scenario's numbers and reconcile them across reads, writes, and storage.

Online reads. 10,000 scoring requests/second, four entities each, is 10,000 × 4 = 40,000 key reads/second at peak. Batched as one multi-get per request, that is 10,000 multi-get calls/second, each fanning to four keys. The p99 budget for the whole vector is 10 ms, so a single key read has to land in low single-digit milliseconds even under fan-out.

Online storage. Count active entities: roughly 50M cards, 20M users, 5M merchants, and 100M devices — about 175M, call it 200M entity rows. Each row holds one model's ~200 features, serialized to about 2 KB. So the hot online footprint is 200M × 2 KB = 400 GB. That fits in a replicated in-memory / SSD-backed key-value cluster; it is a latency problem, not a volume problem.

Offline storage. To guarantee parity we log every served feature vector back to the offline store (more on why in Solutioning). At an average 3,000 transactions/second, that is 3,000 × 2 KB = 6 MB/s, or 6 MB/s × 86,400 s ≈ 500 GB/day. Over a six-month training window, 500 GB × 180 ≈ 90 TB. So the offline store is ~90 TB and scan-bound, three orders of magnitude larger than the online store and a completely different storage profile — columnar files on object storage, not an in-memory KV cluster.

Write / materialization load. Streaming features update on each transaction (~3,000/s average, ~10,000/s peak). Batch materialization bulk-loads the freshest value for ~200M entities into the online store on its schedule; that daily bulk write, not the trickle of streaming updates, is what can overwhelm the online store if left unthrottled.

The two numbers that define the architecture: 400 GB online, 90 TB offline. They cannot be the same store, and keeping them in agreement is the entire game.

API sketch

# Registry (control plane)
POST /api/v1/feature-views
  body: { name, entities:[...], features:[...], transformation, mode: "batch"|"stream", ttl }
POST /api/v1/feature-services            # bundle the feature views a model consumes
  body: { name: "fraud_v3", feature_views:[...] }

# Online serving (hot path, p99 < 10 ms)
POST /api/v1/get-online-features
  body: { feature_service: "fraud_v3",
          entities: { card_id, user_id, merchant_id, device_id } }
  200:  { features: {...}, event_timestamps: {...} }

# Offline / training (async, point-in-time correct)
POST /api/v1/get-historical-features
  body: { feature_service, entity_df: <spine of (entity_ids, event_timestamp, label)> }
  202:  { job_id }                        # → dataset_uri when complete

# Materialization
POST /api/v1/materialize
  body: { feature_view, start_ts, end_ts } # backfill/refresh online store from offline

Solutioning

Start from the two irreconcilable storage profiles and the design falls out. You need a 400 GB latency-optimized online store for the 10 ms read and a 90 TB scan-optimized offline store for training joins, so the first structural decision is dual stores with a materialization bridge between them: the offline store computes and holds history, and a materialization job pushes the latest value per entity into the online store. The reframing that matters here: this is not a caching problem, it is a parity problem. A cache is allowed to be stale or wrong and merely slow you down; these two stores are allowed to differ in freshness but never in meaning, because a model trained on the offline value and serving on the online value has silently learned a lie the moment the two diverge.

That divergence — train/serve skew — is the first defining tradeoff, and the resolution is to attack it at the source of truth rather than by testing outputs. Two feature values drift apart when two code paths compute them: a SQL transformation for the training warehouse and a hand-written Python function in the serving service. So the winning move is a single feature definition compiled to both the batch and streaming engines, plus logging the exact served vector back to the offline store as training data. Logged features are byte-identical to what the model saw, so the training set inherits production reality instead of re-deriving it. The cost is real — logging adds ~500 GB/day to the offline store — but it converts skew from a debugging nightmare into a non-event. The hook: train/serve skew is not a modeling bug, it is a data-lineage bug, and you fix lineage, not the model.

The second defining tradeoff is point-in-time correctness versus join simplicity. Training reconstructs the past, and the naive join — "attach each card's current features to its historical transactions" — leaks the future: it would tell the model a card was flagged fraudulent before the fraud happened. Point-in-time correctness forces an as-of join that, for a transaction at time T, selects only feature values whose event timestamp is ≤ T and whose computed timestamp is also ≤ T, so no value produced with hindsight can leak in. This is what the fraud scenario's "no label leakage" clause demands, and it is expensive — an as-of join over 90 TB is far heavier than an equality join — but there is no shortcut: point-in-time correctness is not a join optimization, it is a leakage-prevention constraint, and getting it wrong produces training metrics that look excellent and a model that fails on day one.

The third tradeoff is freshness versus cost, resolved per feature rather than globally. A card-velocity feature ("transactions by this card in the last five minutes") decays in seconds; computed by a batch job every five minutes it would miss a card-testing attack that fires 200 authorizations in 90 seconds, so it must be a streaming aggregate fresh within ~2 seconds, at the price of a continuously running stream job. A 90-day average ticket size decays over days; recomputing it every few seconds would burn compute for a value that barely moves, so it is a nightly batch feature. The store therefore runs both pipelines, and freshness is a per-feature declaration in the registry — you pay for streaming only where the signal's half-life demands it. The result is a system with one control plane (the registry), two data planes (online and offline) kept in agreement by materialization and served-feature logging, and two compute pipelines (batch and streaming) driven from one definition. The following files take this to components (HLD) and then to schemas, the as-of join algorithm, and the concurrency corners (LLD).