01. Model-Serving Platform — High-Level Design¶

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the solutioning narrative into concrete boxes and the flows between them. Read the architecture top to bottom, then follow an inference request and a version rollout through it, then look at what happens when pieces fail. The split to keep in mind throughout: a data plane that runs forward passes on GPUs as fast as the SLA allows, and a control plane that decides what runs where and when it changes.

Architecture¶

                          ┌──────────────────┐
        client ─────────▶ │  API Gateway      │  auth, quota, per-tenant
                          │  + Rate Limiter   │  admission control
                          └─────────┬─────────┘
                                    │ inference request
                                    ▼
                          ┌──────────────────┐        ┌──────────────────┐
                          │  Inference Router │◀───────│  Model Registry   │
                          │  (version + LB)   │  route │  + Routing Config │
                          └─────────┬─────────┘  table │  (versions, %)    │
                                    │                  └──────────────────┘
              ┌─────────────────────┼─────────────────────┐          ▲
              ▼                     ▼                     ▼          │ traffic %
      ┌───────────────┐     ┌───────────────┐     ┌───────────────┐  │
      │ GPU Replica    │    │ GPU Replica    │    │ GPU Replica    │  │
      │ ┌───────────┐  │    │ ┌───────────┐  │    │ ┌───────────┐  │  │
      │ │ Batch     │  │    │ │ Batch     │  │    │ │ Batch     │  │  │
      │ │ Scheduler │  │    │ │ Scheduler │  │    │ │ Scheduler │  │  │
      │ ├───────────┤  │    │ ├───────────┤  │    │ ├───────────┤  │  │
      │ │ KV-Cache  │  │    │ │ KV-Cache  │  │    │ │ KV-Cache  │  │  │
      │ │ Manager   │  │    │ │ Manager   │  │    │ │ Manager   │  │  │
      │ ├───────────┤  │    │ ├───────────┤  │    │ ├───────────┤  │  │
      │ │ Model v42 │  │    │ │ Model v42 │  │    │ │ Model v43 │  │  │  (canary)
      │ └───────────┘  │    │ └───────────┘  │    │ └───────────┘  │  │
      └───────┬───────┘     └───────┬───────┘     └───────┬───────┘  │
              │  metrics: queue depth, batch fill, KV util, TPOT      │
              └─────────────────────┼──────────────────────┬─────────┘
                                     ▼                      ▼
                          ┌──────────────────┐    ┌──────────────────┐
                          │  Autoscaler       │    │  Metrics / TSDB   │
                          │ (leading signals) │◀───│  + Rollout Judge  │
                          └─────────┬─────────┘    └──────────────────┘
                                    │ scale ±
                                    ▼
                          ┌──────────────────┐
                          │  Model Artifact   │  object store: weights
                          │  Store (S3/GCS)   │  ~26 GB per version
                          └──────────────────┘

Reading it top to bottom: a request enters the API Gateway, which authenticates it, checks the tenant's quota, and applies admission control before letting it in. The Inference Router looks up the target model in the Model Registry, reads the current routing table (which version gets what percentage of traffic), picks a healthy GPU Replica running that version, and hands off the request. Inside each replica, a Batch Scheduler merges the request into the continuously running batch and a KV-Cache Manager allocates its attention state; the model runs and streams tokens back out through the router to the client. Every replica emits queueing and latency telemetry to the Metrics store, which feeds two consumers: the Autoscaler, which grows or shrinks the fleet on leading signals, and the Rollout Judge, which watches a canary version's metrics and decides whether to advance or roll back. New versions are loaded from the Model Artifact Store.

Components¶

API Gateway + Rate Limiter. The front door. It authenticates the caller, maps it to a tenant, and enforces per-tenant quotas before work reaches a GPU — the cheapest place to shed load. Its admission control is the first line of multi-tenant isolation: a tenant over its concurrency budget is queued or 429'd here, never allowed to consume batch slots a paying neighbor needs.

Inference Router. Stateless, and it does two jobs at once: version selection and load balancing. It reads the routing table to decide which version a request belongs to (honoring an explicit X-Version-Pin, else splitting by the canary percentage), then picks among the replicas serving that version by least-loaded (fewest queued/in-flight requests), not round-robin — because request cost varies wildly with generation length and round-robin would pile long generations onto one replica.

Model Registry + Routing Config. The control plane's source of truth: which models exist, which versions each has, where their weights live, and the live traffic split. A rollout is, mechanically, an edit to this config — bump v43 from 1% to 5% — that the router picks up within seconds. Keeping routing as data the router reads (not code it deploys) is what makes rollout and rollback fast.

GPU Replica. One copy of one model version loaded on one or more GPUs, wrapping the two components that carry the design's weight. The Batch Scheduler assembles, every decode step, the set of requests that advance together (continuous batching). The KV-Cache Manager owns the GPU memory that holds each in-flight request's attention state, allocating and freeing it in small pages to avoid fragmentation. These two are detailed in the LLD.

Autoscaler. Watches leading signals — queue depth, batch saturation, KV-cache utilization — rather than lagging GPU utilization, and grows the fleet before p99 breaks. It maintains a warm buffer so there is always spin-up headroom, because a cold replica takes minutes to load 26 GB of weights.

Metrics / TSDB + Rollout Judge. A time-series store for the queueing and latency telemetry every replica emits, plus the automated judge that compares a canary version's error rate and p99 against the incumbent and either signals the next ramp step or triggers rollback.

Model Artifact Store. Object storage holding each version's weights (~26 GB). Cold, cheap, and read only when a replica boots or a new version deploys; never on the request path.

Primary read path (an inference request)¶

The request hits the API Gateway, which authenticates, resolves the tenant, and checks admission control; over-quota tenants are shed here with 429.
The Inference Router resolves the model and version. If the request carries X-Version-Pin: v42, it honors it; otherwise it rolls the dice against the routing table — with v43 at 5%, roughly one request in twenty is sent to a v43 replica.
The router picks the least-loaded replica serving that version and forwards the request.
On the replica, the Batch Scheduler admits the request into the waiting queue; the KV-Cache Manager reserves pages for its prompt. On the next scheduler tick the request's prompt is prefilled (one compute-heavy forward pass over all 500 prompt tokens), producing the first token — this is the TTFT-critical step.
The request then joins the decode batch: each subsequent forward pass advances every in-flight sequence by one token together, and each new token is streamed back to the client over SSE. TPOT is the time of one such step.
When the request emits a stop token or hits max_tokens, the scheduler evicts it and the KV-Cache Manager frees its pages immediately, making room for a queued request. Telemetry (TTFT, TPOT, tokens generated) is emitted to the metrics store off the hot path.

Primary write path (a version rollout)¶

An operator (or CI) calls POST /v1/deployments with version v43 and a canary strategy. The control plane records v43 in the Model Registry pointing at its artifact.
The autoscaler provisions a small number of v43 replicas, which pull ~26 GB from the Artifact Store and load it — the minutes-long cold start, done off to the side while v42 serves 100% of traffic.
Once v43 replicas pass health checks, the control plane edits the routing table to send v43 1% of traffic (10 req/s of the 1,000). The router picks up the change within seconds.
The Rollout Judge watches v43's p99 TTFT/TPOT and error rate against v42's over a bake window (say 10 minutes). If v43 holds the SLA, the config advances to 5%, then 25%, then 100%, provisioning more v43 replicas and retiring v42 ones at each step.
If at any step v43's error rate or p99 regresses past a threshold, the judge (or a human via /rollback) resets the routing table to 0% v43 — instant, because it is a config edit, not a redeploy — and the fleet returns to all-v42.

Storage choices¶

Model weights: object store (S3/GCS). Large (~26 GB/version), immutable once published, read-mostly at boot. Object storage is cheap, durable, and versioned; the ~minutes to pull weights is why cold start is a capacity problem, so replicas cache weights on local NVMe to make re-launch faster.
Routing config + version metadata: a small strongly-consistent store (etcd / a relational table). Tiny but hot on reads by every router and read-your-writes on rollout edits — the router must see a rollback immediately. Consistency and low-latency reads matter far more than volume here.
KV cache: GPU HBM, managed in pages (ephemeral). The per-request attention state. It lives and dies with the request and never persists; the only durability question is whether to offload cold prefixes to CPU RAM for prefix reuse, covered in the LLD.
Metrics: time-series store. High-cardinality queueing and latency series, queried by aggregation over short windows for autoscaling and rollout decisions. A TSDB (Prometheus/VictoriaMetrics/Cortex) fits; keeping it off the serving store protects the request path from telemetry load.

Scaling¶

Request path. Scale out by adding GPU replicas; the router spreads load across them by least-loaded. The natural unit of scale is the replica (one model copy), and because the router is stateless and reads a shared routing table, adding replicas is a matter of registering them. Going from 1,000 to 2,000 req/s roughly doubles the fleet from ~150 to ~300 GPUs at the same 70% utilization target — GPU count scales linearly with request rate, which is exactly why per-GPU throughput (the batching knob) is the cost lever.

Batch scheduling. Within a replica, throughput scales with batch size until either the KV cache fills or TPOT hits the SLA ceiling. The KV-cache ceiling is the usual binding one: at ~0.8 MB/token and ~13 GB usable, a replica holds ~23 full-length sequences naively, or ~40–50 with paged allocation eliminating fragmentation — nearly doubling per-GPU throughput and halving the fleet needed for the same load. That is the single highest-leverage number in the system.

Autoscaling. Grow on leading signals: sustained queue depth above a threshold, or KV-cache utilization past ~85%, means the next few seconds will breach SLA — scale now, before p99 moves. Because a new replica takes minutes to warm, the autoscaler keeps a warm buffer (say 10% extra replicas) so it always has ready capacity while it launches more. Scale-down is deliberately slower and hysteretic to avoid thrashing a fleet whose spin-up is expensive.

Hot model / hot tenant. A single popular model or a bursting tenant is absorbed by pooling requests across all replicas of that version and by admission control at the gateway. Unlike a stateless web tier, you cannot instantly shard a hot model onto more GPUs — each new replica pays the cold-start tax — so the warm buffer and predictive scaling do the real work here.

Operational signals¶

The healthy signal is batch fill relative to the TPOT ceiling: replicas running near their max batch while TPOT sits just under the SLA means the fleet is both busy and safe — the ideal. The first metric to degrade under trouble is queue depth (time-in-queue before prefill), which climbs the instant arrival rate outpaces capacity and shows up in TTFT long before anything else; it is the autoscaler's trigger for exactly this reason. The misleading metric is GPU utilization percent — it can read 95% while the platform is quietly failing its SLA (a huge batch keeps the GPU pegged but pushes TPOT over the cliff), or read a comfortable 60% while requests pile up in queue waiting for KV-cache pages to free; utilization tells you the GPU is busy, not that requests are being served well. The graph an experienced operator opens first during an incident is the KV-cache utilization curve per replica: when it pins near 100%, the scheduler cannot admit new requests, queue depth backs up, and TTFT breaks — that curve is usually the root cause a latency page is really about.

Failure modes and resilience¶

A GPU replica dies mid-generation. Every in-flight sequence on it is lost — their KV cache was in that GPU's memory and is unrecoverable. Mitigation: the router detects the health-check failure, stops routing to it, and clients retry idempotently against another replica (generation restarts from the prompt). Keeping requests below a max token budget bounds how much work a single failure can throw away.
A bad new version regresses under the canary. This is the threaded scenario's danger: v43 goes to 1% (10 of the 1,000 req/s) and its p99 TPOT jumps from 22 ms to 60 ms, or its error rate spikes. Because only 1% of traffic is exposed and rollout is a config edit, the Rollout Judge trips and resets v43 to 0% within seconds, and the other 990 req/s never saw it. The blast radius was capped at 10 req/s by design — the entire point of canarying at 1% first.
A traffic spike outruns cold start. Load doubles in 30 seconds but a replica needs 3 minutes to warm. Reactive scaling loses the race and requests time out. Mitigation: the warm buffer absorbs the first minutes while predictive scaling (or a scheduled pre-warm before a known event) launches replicas ahead of demand; the gateway sheds excess with 429 rather than accepting requests it cannot serve within SLA.
KV-cache exhaustion / preemption storm. When memory fills, the scheduler must preempt in-flight sequences (evict their KV cache and re-prefill them later) to make progress, and under sustained overload this thrashes — sequences are evicted and recomputed repeatedly, collapsing throughput. Mitigation: cap admitted concurrency below the preemption point and shed at the gateway rather than admitting into a thrashing scheduler.
Noisy-neighbor tenant. One tenant floods long-generation requests and consumes batch slots and KV pages. Mitigation: per-tenant admission quotas at the gateway and per-tenant caps on concurrent batch slots and KV pages in the scheduler, so the flood fills that tenant's budget and queues rather than starving others.
Router or registry outage. If the routing config store is unavailable, routers serve the last-known-good routing table from local cache and keep serving inference; only rollouts (which need to write config) pause. The request path is designed to survive a control-plane outage read-only.

Where this shows up in production¶

OpenAI / Anthropic inference endpoints — front a fleet of GPU replicas with continuous batching and version-pinned routing, streaming tokens over SSE, exactly the data-plane shape here.
vLLM — the open-source engine that popularized PagedAttention, managing KV cache in fixed pages to cut fragmentation waste from ~60–80% to under 4%, which is the batch-doubling lever this design leans on.
NVIDIA Triton Inference Server — the multi-framework serving layer whose dynamic batching is the classic-ML ancestor of continuous batching, and whose model-repository/versioning maps to our registry.
AWS SageMaker endpoints — production canary/linear traffic-shifting between model variants behind one endpoint, the managed version of our Rollout Judge and routing table.
Google Vertex AI / KServe — declarative model deployments with traffic splits across versions and autoscaling (including scale-to-zero for cold models), the control-plane pattern here.
Ray Serve — composes replicas and autoscaling for Python model serving, treating the replica as the unit of scale as we do.
Hugging Face TGI (Text Generation Inference) — continuous batching plus token streaming for open LLMs, the reference implementation of the scheduler-per-replica model.
Meta / internal recommendation serving — the multi-tenant, latency-SLA'd accelerator-sharing problem at scale, where admission control and fair queueing are the isolation mechanism rather than dedicated hardware.