00. Design a Model-Serving / Inference Platform¶

~20 min read · Level: advanced · Part 1 of 4 (Overview → HLD → LLD → Q&A)

Problem¶

A model-serving platform takes a trained model and puts it behind an API that turns inputs into predictions, holding a latency SLA while a fleet of expensive accelerators stays busy. This is the system behind an OpenAI or Anthropic inference endpoint, an AWS SageMaker or Google Vertex endpoint, and the open-source stack (vLLM, TGI, Triton) that companies run on their own GPUs. A client sends a prompt, the platform routes it to a GPU holding the right model version, runs the forward pass, and streams tokens back. The job sounds like "wrap a model in HTTP," and for one request on one GPU it is. What turns it into a system-design problem is that the accelerator is the scarcest and most expensive resource in the building, the work is bursty and variable-length, and the only way to make a GPU economical is to run many requests through it at once — which fights directly against the latency you promised each of them.

The tension is sharper for large language models than for classic ML because generation is autoregressive: a request does not finish in one forward pass, it dribbles out one token at a time over seconds, holding GPU memory the whole time. So the platform is not scheduling short, uniform tasks; it is juggling dozens of long-lived, unequal-length streams on each device, deciding every few milliseconds which ones advance together.

To keep the reasoning concrete, thread one scenario through the whole design: serve a 13-billion-parameter chat model at 1,000 requests/second on a GPU fleet, holding a p99 latency SLA, while rolling out a new model version safely. The average request carries ~500 prompt tokens in and generates ~200 tokens out. The SLA is on the streaming experience — p99 time-to-first-token ≤ 500 ms and p99 time-per-output-token ≤ 30 ms — not on total wall-clock, because the response streams. That one workload, and the tension between packing GPUs full and keeping those two tail numbers under their ceilings, will test every decision below.

Functional requirements¶

Inference: given a model name and input, run the forward pass and return the output; for LLMs, stream tokens as they are generated.
Model versioning: register multiple versions of a model, address a specific version or a moving alias like chat-model@stable, and serve several versions concurrently.
Safe rollout: shift traffic from an old version to a new one gradually (canary → ramp → full), with a fast rollback.
Autoscaling: grow and shrink the GPU fleet with load so we neither drop requests at peak nor pay for idle accelerators at 3 a.m.
Multi-tenancy: serve multiple models and multiple callers on shared hardware without one tenant's burst starving another.

De-scoped for this round, and worth saying out loud so the interviewer hears a choice rather than a gap: model training and fine-tuning pipelines, the feature store and data plane that feed classic ML models, prompt/response content moderation, and per-token billing. These are real neighboring systems, but they sit beside the serving core and do not change its shape.

Non-functional requirements¶

The dominant constraint is holding p99 latency while keeping a bounded, very expensive GPU fleet near saturation — the batching knob trades these two against each other directly, and every other decision bends around that trade.

Latency (tail, not mean): the p99 TTFT and TPOT ceilings above must hold under load and during a rollout. The mean is nearly useless here; a platform can show a healthy average while a saturated queue quietly pushes p99 past its cliff.
Throughput per dollar: GPUs dominate cost. At ~\(2.50/GPU-hour, a 150-GPU fleet is ~\)270k/month, so a 20% drop in per-GPU throughput is real money, not a rounding error. Utilization is the cost lever.
Availability: the endpoint must stay up and must keep honoring version pins through a rollout; a bad new version must never take down the old one.
Isolation: a tenant sending a flood of long-generation requests must not blow another tenant's p99. Fairness is a first-class requirement, not a nice-to-have.
Elasticity vs. cold start: capacity must track load, but a cold GPU replica takes minutes to become useful (download and load ~26 GB of weights), so we cannot simply scale reactively at request time.

Scale estimation¶

Take the scenario at face value: 1,000 requests/second, each ~500 tokens in and ~200 tokens out.

Start with the model's memory footprint, because that sets everything else. A 13B model in fp16 is 13e9 params × 2 bytes ≈ 26 GB of weights. On a 40 GB A100 that leaves roughly 12–14 GB for the KV cache (the per-token attention state every in-flight request holds). The KV cache costs about 2 × n_layers × hidden_dim × 2 bytes ≈ 2 × 40 × 5120 × 2 ≈ 0.8 MB per token. A typical 700-token request (500 + 200) therefore holds ~560 MB of KV cache while it lives. Divide: ~13 GB / 0.56 GB ≈ 23 concurrent requests fit on one GPU as a hard memory ceiling. That ceiling — not compute — is what caps batch size, and it is the number the whole design pushes against.

Now capacity. With continuous batching keeping ~40–50 requests in flight per GPU (paged KV cache lets us pack past the naive 23 by nearly eliminating fragmentation, covered in the LLD) and each request completing in ~4–5 s of GPU-shared time, one GPU sustains roughly 10 completed requests/second. So 1,000 / 10 = 100 GPUs at 100% utilization. But you cannot run a latency-SLA service at 100% — queueing theory says wait time explodes as utilization approaches 1. Hold the fleet near 70% and you need 1,000 / (10 × 0.70) ≈ 143, so provision ~150 GPUs. That headroom is not waste; it is the p99 SLA bought in hardware.

Token throughput sanity-checks the fleet. Decode demand is 1,000 req/s × 200 out-tokens = 200,000 output tokens/second; at ~2,000 output tokens/s/GPU of decode throughput that is 100 GPUs, matching the request-rate estimate. Prefill demand is 1,000 × 500 = 500,000 prompt tokens/second, but prefill is compute-bound and runs several times faster per token than decode, so it is interleaved on the same GPUs rather than needing its own fleet. The two estimates agreeing at ~100 GPUs saturated is the reconciliation.

Storage is trivial next to compute: each model version is ~26 GB in an object store, and even fifty versions is ~1.3 TB — pocket change. The interesting scaling here is entirely on the accelerator and the scheduler, never on disk.

API sketch¶

POST /v1/models/{model}/infer
  headers: { "X-Version-Pin"?: "v42" }          # else routed by alias
  body:    { "input": "...", "max_tokens": 200, "stream": true }
  200:     text/event-stream                     # tokens streamed as SSE
             data: {"token": "Hello", "index": 0}
             ...
             data: {"finish_reason": "stop", "usage": {...}}
  429:      over quota / tenant rate limited
  503:      no healthy replica for this version

POST /v1/deployments                             # control plane
  body:    { "model": "chat-model", "version": "v43", "strategy": "canary",
             "steps": [ {"pct": 1}, {"pct": 5}, {"pct": 25}, {"pct": 100} ] }
  202:     { "deployment_id": "d-991", "state": "ramping" }

POST /v1/deployments/{id}/rollback
  202:     { "state": "rolling_back" }

GET  /v1/models/{model}/versions
  200:     [ { "version": "v42", "traffic_pct": 95, "state": "stable" },
             { "version": "v43", "traffic_pct": 5,  "state": "canary" } ]

Solutioning¶

Start from the one fact that reorganizes everything: the GPU is idle most of the time unless you keep it full, and keeping it full means running many requests through it in one forward pass. So the platform's heart is not a request router — it is a batch scheduler living next to each GPU, continuously assembling the set of requests that will take the next step together. The reframing to carry into the room: serving an LLM at 1,000 req/s is not a request-routing problem; it is a batch-scheduling problem. Routing decides which replica; the scheduler decides who advances together, and that second decision is where both the throughput and the p99 are won or lost.

The first defining tradeoff is batch size versus latency, and it moves hard numbers. A larger batch amortizes the weight-loading cost of each forward pass across more requests, so per-GPU throughput climbs and the fleet shrinks — but every request in the batch shares one step, so a bigger batch means a longer step, and TPOT rises with it. Concretely: a batch of ~45 holds TPOT near 22 ms and needs ~100 GPUs; pushing the batch to ~80 to chase throughput drives TPOT to ~45 ms and blows the 30 ms SLA, while cutting the batch to ~20 to protect latency doubles the fleet toward 200+ GPUs and the monthly bill with it. The resolution is not a fixed batch size but continuous batching: requests join and leave the running batch every decode step instead of waiting for a batch window to fill and drain, so newcomers start within a step or two (protecting TTFT) and finished requests free their slot immediately (protecting throughput). The scheduler caps the batch at the point where TPOT still clears the SLA, and no higher.

The second tradeoff is cold start versus cost. A cold replica needs minutes to pull and load 26 GB of weights, so you cannot answer a traffic spike by launching GPUs when the spike arrives — the requests time out long before the replica is ready. The honest reframing: cold start is not a latency problem you can autoscale away; it is a capacity problem you pre-warm around. So the platform scales on a leading signal — queue depth and batch saturation, not lagging GPU utilization — and keeps a small warm buffer of extra replicas so it always has a few seconds of headroom while new ones spin up. That warm buffer costs idle-GPU money, and sizing it is the explicit dial between "never drop a request" and "never pay for a dark GPU."

The third tradeoff is multi-tenant isolation versus utilization. The cheapest way to run GPUs is to pool every tenant's traffic onto shared replicas so batches stay full; the safest way is to give each tenant dedicated GPUs so no one can hurt anyone else — and dedicated GPUs sit half-empty. The middle path is shared replicas with per-tenant admission control and fair queueing: everyone shares the hardware, but the scheduler bounds how many batch slots and how much KV-cache any one tenant can hold at once, so a tenant firing 500 long-generation requests fills its own quota and queues, rather than evicting a neighbor's in-flight sequences. Isolation becomes a scheduling policy, not a hardware partition.

The result is a system whose data plane is a fleet of GPU replicas each fronted by a continuous-batching scheduler and a KV-cache manager, whose control plane owns the model registry and drives version rollouts by moving traffic percentages, and whose autoscaler watches queueing signals to grow the fleet ahead of demand. The next file turns these into boxes and flows; the one after pins down the scheduler, the KV-cache, the autoscaler, and the rollout machinery down to the algorithm.