01. Feature Store + Training Pipeline — High-Level Design¶

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the solutioning narrative into boxes and the flows between them. Read the architecture top to bottom, then follow a feature from event to training row and from event to a 10 ms serving read, then look at what happens when the pieces drift or fail.

Architecture¶

   raw events (transactions, clicks, chargebacks)
            │
            ▼
   ┌─────────────────────┐        ┌──────────────────────┐
   │  Streaming pipeline  │        │   Batch pipeline      │
   │  (Flink: velocity,   │        │   (Spark: long-window │
   │   5-min counts)      │        │    aggregates, daily) │
   └──────────┬───────────┘        └───────────┬──────────┘
              │  writes                          │  writes
              ▼                                  ▼
   ┌───────────────────────────────────────────────────────┐
   │            Offline store (system of record)            │
   │   columnar files on object storage, ~90 TB, 6 months   │
   │   feature logs + computed history, event/created ts    │
   └───────┬──────────────────────────────────┬────────────┘
           │  materialize (latest per entity)  │  point-in-time join
           ▼                                   ▼
   ┌──────────────────┐                ┌──────────────────────┐
   │  Online store     │                │  PIT join engine      │
   │  (KV, ~400 GB,    │                │  (as-of join → training│
   │   p99 < 10 ms)    │                │   dataset)            │
   └────────┬─────────┘                └───────────┬──────────┘
            │                                       │
            ▼                                       ▼
   ┌──────────────────┐                       training job
   │  Serving API      │◀── fraud scoring service (10k req/s)
   │  (multiget +      │──▶ logs served vector back to offline
   │   freshness check)│
   └──────────────────┘

   ┌──────────────────┐   defines & governs every box above
   │  Feature Registry │   (feature views, transformations,
   │  (Postgres)       │    freshness, versions, lineage)
   └──────────────────┘

Read it as two data planes fed by two pipelines and governed by one registry. Raw events flow into a streaming pipeline (fresh aggregates within seconds) and a batch pipeline (heavy long-window aggregates, daily). Both write to the offline store, which is the system of record and holds ~90 TB of history. From there the data forks: a materialization job pushes the latest value per entity into the ~400 GB online store for serving, and a point-in-time join engine reads the full history to build leak-free training datasets. The serving API answers the fraud model's reads in under 10 ms and logs each served vector back to the offline store so training sees exactly what production saw. The registry sits to the side but governs everything: no feature exists until it is defined there, and its definition is what compiles to both pipelines.

Components¶

Feature Registry. The control plane and single source of definitions. A feature view names its entities, the features it produces, the transformation that computes them, its mode (batch or stream), and its TTL. Because both pipelines compile from this one definition, the registry is the mechanism that prevents two divergent code paths — the root cause of skew. It also carries versioning and lineage, so you can answer "which model was trained on which version of this feature."

Streaming pipeline. A stateful stream processor (Flink or equivalent) that maintains rolling aggregates — the card's 5-minute transaction count, the device's failed-auth count this hour. It updates the online store within seconds and appends to the offline log. This exists for features whose signal decays in seconds; it is the expensive pipeline, run continuously, so only features that need it live here.

Batch pipeline. A scheduled job (Spark/SQL over the warehouse) that computes long-window features — 90-day average ticket, lifetime chargeback rate — over full history. It writes to the offline store and feeds materialization. It is cheap per feature but coarse in freshness, so it owns slow-moving signals.

Offline store. The system of record: columnar files (Parquet/Delta) on object storage, or a warehouse. It holds all historical feature values plus logged served vectors, partitioned by date and entity. Its access pattern is large scans and as-of joins, not point lookups, which is why it is columnar and not a KV store.

Online store. A low-latency key-value store (Redis / DynamoDB / Cassandra) holding only the latest value per entity per feature view — ~400 GB. Keyed by entity, its only query is a point multiget. It is not a source of truth; a cold or lost online store is rebuilt by re-materializing from the offline store.

Materialization engine. Moves the latest computed value per entity from offline to online. It is what keeps the two stores in agreement, and its lag is the freshness of batch features on the serving path. It must be throttled: bulk-loading ~200M entity rows can otherwise saturate the online store and blow the 10 ms SLA for live reads.

Point-in-time (PIT) join engine. Takes a spine of (entity_ids, event_timestamp, label) and produces a training dataset by as-of joining each row to the feature values known just before its timestamp. This is where leakage is prevented or introduced; it is the offline read path's signature component.

Serving API. Answers get-online-features in under 10 ms via a multiget against the online store, assembles the vector, checks freshness against TTL, logs the served vector, and returns. It is stateless and scales horizontally.

Primary write path (event to stored feature)¶

A raw event — a transaction, a chargeback, a device signal — lands on the event bus.
The streaming pipeline consumes it and updates any streaming aggregate it feeds. Incrementing the card's 5-minute count writes the new value, tagged with the event's timestamp, to the online store within ~2 seconds, and appends the same value to the offline log.
In parallel, the event is durably retained so the batch pipeline can, on its schedule, recompute long-window features over the full history and write them to the offline store.
The materialization engine periodically reads the latest batch-computed value per entity from the offline store and upserts it into the online store — but only if the incoming value's event timestamp is newer than what the online store already holds, so a slow batch load can never overwrite a fresher streaming value.
The offline store thus accumulates both computed history and, from the serving path, logged served vectors — the raw material for point-in-time-correct training.

Primary read path (two of them)¶

Online read (the 10 ms path). The fraud scoring service calls get-online-features with the card, user, merchant, and device ids. The serving API issues one multiget fanning to four keys in the online store, assembles the ~200-feature / ~2 KB vector, checks each value's event timestamp against its TTL (a value older than its TTL is served as null, matching how training treats an expired value), fires an asynchronous log of the served vector to the offline store, and returns. The whole path is one multiget plus in-process assembly, which is what keeps p99 under 10 ms.

Offline read (the training path). A data scientist submits an entity spine — for the fraud model, ~200M rows of (card_id, user_id, merchant_id, device_id, transaction_timestamp, label) — to get-historical-features. The PIT join engine, for each row, as-of joins to the feature values whose event timestamp and computed timestamp are both ≤ the transaction timestamp, takes the latest per feature, respects TTL, and writes the resulting dataframe to a dataset URI. This is a heavy distributed scan over 90 TB, run as an async job, not a synchronous call.

Storage choices¶

Online store: replicated KV (Redis / DynamoDB / Cassandra). The access pattern is a point multiget by entity key under a 10 ms budget. A KV store with in-memory or SSD-backed reads and TTL support fits exactly. It shards cleanly on the entity id, which is uniform, so no shard becomes a key-skew hotspot.
Offline store: columnar files on object storage (Parquet/Delta) or a warehouse. The access pattern is full-history scans and as-of joins over 90 TB. Columnar layout, date/entity partitioning, and predicate pushdown make the PIT join tractable; a KV store could not scan this way.
Registry: relational (Postgres). Definitions, versions, and lineage are small, highly relational, and read on the control path, not the hot path — a classic small relational workload.
Streaming state: the stream processor's own state store (RocksDB-backed). Rolling-window aggregates need local, fault-tolerant state with checkpointing; keeping it in the processor avoids a round trip per event.

Scaling¶

Online read path. 40,000 key reads/second shard across the KV cluster by entity id. With, say, 8 shards that is ~5,000 reads/second/shard, comfortably within a single node's budget at sub-millisecond reads; because the key is a uniform entity id, adding shards scales reads linearly with no hotspot. Batching four keys into one multiget per request cuts round trips 4:1, which is the difference between a 10 ms and a 25 ms p99 under fan-out.

Materialization. The daily bulk load of ~200M entity rows is the write-side risk. Left unthrottled it competes with live reads and pushes p99 past 10 ms; the lever is to rate-limit the load and materialize incrementally (only entities whose values changed), turning a 200M-row cliff into a smooth trickle that the online store absorbs alongside serving.

Offline / training. The 90 TB as-of join scales by partition pruning — a training run over one month reads ~15 TB, not 90 — and by distributing the join across a Spark cluster. Cost scales with spine size and window length, which is why teams sample the spine (200M rows, not billions) rather than joining the entire event history.

Streaming. Throughput scales by partitioning the stream on entity id so each key's aggregate lives on one worker; the ~10,000 events/second peak is modest, but a viral card-testing burst concentrated on a few cards is the skew case, handled by keyed state and backpressure rather than by adding shards.

Operational signals¶

The healthy signal is feature freshness lag — the age of the value in the online store versus the latest event — which should sit within each feature's declared SLA (seconds for streaming, hours for batch) and barely move. The first metric to degrade under trouble is online read p99, which climbs when a materialization bulk load is competing with live reads or a shard is hot. The misleading metric is serving availability and latency: the API can return 200s at a healthy 8 ms while every value it serves is twelve hours stale, because a frozen pipeline breaks freshness, not serving — the store answers instantly with old data. Watch freshness lag, not the success rate. The graph an experienced operator opens first during an incident is the per-feature-view materialization / streaming lag, because that single panel distinguishes "serving is slow" (a latency problem) from "serving is fast but wrong" (a freshness problem), and the second is the one that silently degrades the fraud model.

Failure modes and resilience¶

Streaming pipeline stalls (the silent failure, and the threaded one). If the stream job feeding the card-velocity feature freezes, the online store keeps answering in 8 ms with 200s — but the value is frozen at its last update. A card-testing attack firing 200 authorizations in 90 seconds is invisible to the model because the "5-minute count" never moves past the pre-stall value, and fraud sails through. Serving latency and availability look perfect the entire time. Detection is freshness lag, not error rate; the recovery lever is a freshness-based circuit that flags or degrades the feature when its lag exceeds SLA, and streaming checkpoints that let the job resume from the last committed offset without replaying from zero.
Materialization stalls. Batch features on the serving path go stale. For slow-moving features (90-day average) a few hours of staleness is tolerable; the resilience choice is per-feature TTL, so a value older than its TTL is served as null rather than as confidently-stale, and the model handles null the same way it did in training.
Online store outage. The fraud model cannot read features and must choose fail-open (approve unscored, accept fraud loss) or fail-closed (decline, accept revenue loss). Neither is free; the design decision is to make it explicit per feature service and to keep a replicated online store with fast failover so the choice is rarely exercised. The offline store is untouched, so re-materialization rebuilds a lost online store.
Train/serve skew from divergent definitions. If someone hand-writes a serving transformation that diverges from the training SQL, the model degrades with no error anywhere. Prevention is structural — one definition compiled to both paths — and detection is a skew monitor that recomputes logged served vectors offline and diffs them against what was served.
Point-in-time leakage. A join bug that admits a future value produces training metrics that look excellent and a model that fails immediately in production. This is caught not in serving but in offline validation: hold-out backtests and comparing offline-predicted to online-observed performance.
Late-arriving data. A chargeback label or a backfilled feature value arrives with a computed timestamp after its event timestamp. Handled by tracking event and created timestamps separately, so the PIT join can exclude values that were not yet knowable at the training row's instant.

Where this shows up in production¶

Uber (Michelangelo / Palette) — pioneered the dual-store pattern, an offline store (Hive) for training and an online store (Cassandra) for serving, and named the parity problem the whole industry now designs around.
Airbnb (Zipline) — made point-in-time-correct training-set generation a first-class API precisely to kill the label-leakage class of bug at the platform level rather than per team.
Tecton — compiles one feature definition to both batch and streaming engines and logs served features, the two mechanisms that turn skew from a debugging problem into a structural non-issue.
Feast — the open-source reference architecture: a registry, an offline store, an online store, and a materialization step that moves latest values between them.
Netflix — practices "log the features you served," treating the logged serving-time vector as the authoritative training input so training cannot drift from serving.
Stripe (Radar) — reads real-time velocity and reputation features at authorization time under a hard latency budget, the exact fraud-scoring shape threaded through this study.
DoorDash — runs a Redis-backed online store serving ML features at high QPS with tight tail-latency targets for real-time ranking and dispatch.
LinkedIn (Feathr) — built point-in-time joins over very large offline datasets as the core of its training-data generation.
Meta — runs streaming feature aggregation at scale for integrity and ranking, where velocity features must reflect events within seconds.
Robinhood and other real-time-decision shops — make the fail-open vs fail-closed choice on online-store outage an explicit, per-decision policy rather than an accident of a timeout.