03. Model-Serving Platform — Interview Q&A

~15 min read · Part 4 of 4 (Overview → HLD → LLD → Q&A)

These are the questions an interviewer actually asks once the diagram is on the board. Each answer is the strong version, followed by the wrong answer that quietly sinks candidates.

Q1. Why is batching the heart of this system, and not routing? Because a GPU is idle most of a forward pass unless many requests share it — the weight matrices get loaded from HBM once per step regardless of how many sequences ride along, so one request wastes the same memory bandwidth as forty. Batching is what turns a $2.50/hour accelerator from serving ~1 req/s into serving ~10. Routing only decides which replica; the batch scheduler decides who advances together, and that decision sets both throughput and p99. The memory hook: serving an LLM at 1,000 req/s is not a request-routing problem, it is a batch-scheduling problem. Common wrong answer to avoid: "Put a load balancer in front of GPU replicas and scale horizontally." That is necessary but not the point; a perfectly balanced fleet with batch size 1 still needs 10× the GPUs and burns 10× the money.

Q2. How do you serve 1,000 req/s and how many GPUs does it take? Size from the model's memory first. A 13B model is ~26 GB in fp16, leaving ~13 GB on a 40 GB A100 for KV cache at ~0.8 MB/token; a 700-token request holds ~560 MB, so ~23 sequences fit naively, ~40–50 with paged allocation. At batch ~45 and ~4–5 s per request, one GPU sustains ~10 req/s, so 1,000 req/s needs ~100 GPUs at 100% utilization. You cannot run a latency SLA at 100%, so target ~70% and provision ~150 GPUs. The token math checks it: 1,000 × 200 output tokens = 200k tokens/s ÷ ~2,000 tokens/s/GPU ≈ 100 GPUs saturated. Common wrong answer to avoid: "A few big GPUs should handle it." Without deriving the KV-cache ceiling and the utilization headroom, the number is a guess, and the guess is always too low.

Q3. Explain the batch-size versus latency tradeoff with numbers. A bigger batch amortizes the per-step weight load across more sequences, so per-GPU throughput rises and the fleet shrinks — but every sequence in the batch shares one forward pass, so a bigger batch means a longer step, and TPOT rises with it. Concretely: batch ~45 holds TPOT near 22 ms and needs ~100 GPUs saturated; pushing to ~80 to chase throughput drives TPOT to ~45 ms and blows the 30 ms SLA; cutting to ~20 to protect latency doubles the fleet toward 200+ GPUs. The resolution is continuous batching with the batch capped where measured step time still clears the SLA — highest throughput that keeps p99 legal, and no higher. Common wrong answer to avoid: "Use the largest batch the GPU memory allows." Memory is not the binding limit for latency; step time is, and the largest batch that fits will breach TPOT long before it runs out of memory.

Q4. What is continuous batching and why not static batching? Static batching collects N requests, runs them to completion together, then takes the next N. It wastes the GPU whenever sequences finish at different lengths — a batch generating 10 and 200 tokens leaves the short one's slot idle for 190 steps — and it makes a fresh arrival wait for the whole batch to drain, wrecking TTFT. Continuous batching treats the running batch as a set edited every decode step: finished sequences are evicted and their KV pages freed immediately, and queued arrivals are admitted on the very next step. Utilization stays high because slots refill instantly, and TTFT stays low because a newcomer waits a step or two, not a whole batch. Common wrong answer to avoid: "Batch requests in a 50 ms time window, then run them together." That is static/dynamic batching; it adds a fixed queueing delay to every request and idles the GPU on length skew.

Q5. What is the KV cache and why does it bound everything? During generation each token attends to all previous tokens, so the platform caches the key/value tensors of every prior token rather than recomputing them — that is the KV cache, and it grows with sequence length and batch size. At ~0.8 MB/token for this model, it fills GPU memory fast: ~13 GB usable holds only ~23 full requests. It is the true ceiling on batch size, ahead of compute. That is why PagedAttention matters: managing the cache in 16-token pages instead of one contiguous per-request buffer cuts fragmentation waste from ~60–80% to under 4%, packing ~40–50 sequences into the same memory — nearly doubling per-GPU throughput and halving the fleet for the same load. Common wrong answer to avoid: "Just recompute attention each step to save memory." That trades a memory problem for a quadratic-compute problem and makes generation dramatically slower; the KV cache exists precisely to avoid it.

Q6. Walk through rolling out a new version safely at 1,000 req/s. Warm v43 replicas off to the side (they take ~3 minutes to pull and load 26 GB) while v42 serves 100%. Then edit the routing table to send v43 1% — ~10 of the 1,000 req/s — and bake for ~10 minutes while a judge compares v43's p99 TPOT and error rate against v42 and against the absolute SLA. If v43 holds, ramp to 5%, 25%, 100%; if it regresses — say its p99 TPOT measures 41 ms against a 30 ms SLA — reset its traffic to 0%, which takes effect in seconds because routing is a config edit, not a redeploy. The blast radius was capped at 10 req/s; the other 990 never moved. Canarying at 1% turns a platform-wide incident into a dashboard footnote. Common wrong answer to avoid: "Deploy v43 and cut over." A big-bang cutover exposes all 1,000 req/s to a bad version at once, and rollback becomes a minutes-long redeploy while every request suffers.

Q7. What signal do you autoscale on, and why not GPU utilization? Scale on leading queueing signals — sustained queue depth and KV-cache utilization past ~85% — because they predict an SLA breach before latency actually moves. GPU utilization is misleading: it can read 95% while a giant batch pushes TPOT over the cliff, or read 60% while requests pile up waiting for KV pages to free. Utilization says the GPU is busy, not that requests are being served within SLA. And because a cold replica needs minutes to load weights, reactive scaling loses the race against a spike, so you keep a warm buffer (say 10% spare replicas) and scale up early, scale down slowly with hysteresis. Common wrong answer to avoid: "Autoscale on GPU utilization like CPU utilization." It is the classic trap — the number correlates poorly with SLA health and gives no early warning, and reactive scaling can never win against a multi-minute cold start.

Q8. Cold start takes minutes — how do you handle a sudden traffic spike? Accept that you cannot launch GPUs when the spike arrives; a replica needs minutes to pull and load 26 GB. So this is a capacity problem, not a latency problem: keep a warm buffer of ready replicas sized to absorb the first few minutes, scale on leading signals so launches start before p99 moves, pre-warm ahead of known events (a product launch, a marketing push), and cache weights on local NVMe so re-launch is faster than a cold pull. When demand still outruns capacity, shed at the gateway with 429 rather than admitting requests you cannot serve within SLA. The memory hook: cold start is not a latency problem you autoscale away, it is a capacity problem you pre-warm around. Common wrong answer to avoid: "Autoscale reactively when load rises." By the time a new replica finishes loading, the spike's requests have already timed out.

Q9. How do you stop one tenant from wrecking another's p99 on shared GPUs? Enforce isolation as a scheduling policy, not a hardware partition. At the gateway, per-tenant quotas and admission control shed a tenant's excess before it reaches a GPU. In the scheduler, per-tenant caps on batch slots and KV-cache pages mean a tenant firing 500 long generations fills its own budget and its extra requests rotate to the back of the queue, rather than evicting a neighbor's in-flight sequences. Dedicated GPUs per tenant would guarantee isolation but leave hardware half-idle and blow the cost model; fair queueing on shared replicas keeps utilization high while bounding blast radius. Common wrong answer to avoid: "Give each tenant their own GPUs." Clean isolation, terrible economics — you are back to paying for idle accelerators, which is the whole thing this platform exists to avoid.

Q10. A GPU replica dies mid-generation. What happens? Every in-flight sequence on it is lost, because their KV cache lived in that GPU's memory and is not recoverable. The router detects the health-check failure, stops routing there, and clients retry idempotently against another replica — generation restarts from the prompt. The contract is "a valid completion," not "the identical completion," since sampling is random, so a retry may differ; a client-supplied request_id keeps a racing retry from double-counting usage. Capping max_tokens bounds how much work one failure discards. You do not try to replicate KV cache across GPUs — it is huge and ephemeral, and re-prefilling from the prompt is cheaper than paying to mirror it continuously. Common wrong answer to avoid: "Replicate the KV cache so we can fail over mid-stream." The cache is hundreds of MB per request and changes every token; mirroring it would cost more bandwidth than the inference itself.

Q11. Why keep routing as config data instead of deploying it in code? Because rollback has to be instant. If the traffic split is a row the router re-reads within seconds, resetting a bad canary to 0% is a config write that takes effect almost immediately. If the split were baked into a deployed artifact, rollback would mean a redeploy — minutes during which a broken version keeps taking traffic. Routers watch the config store rather than caching on a fixed TTL, so a rollback propagates fast, and they fall back to last-known-good only if the store itself is unreachable, so a control-plane outage degrades to "no new rollouts," never "inference stops." Common wrong answer to avoid: "Bake the traffic percentages into the deployment." That couples rollback speed to deploy speed and turns a 5-second recovery into a multi-minute outage.

Q12. Why is p99 the SLA and not average latency, and where does the mean lie to you? Because tail latency is what users feel and what a saturated queue reveals first. As the fleet approaches capacity, queueing theory says wait time explodes — the mean barely moves while p99 walks off a cliff, because most requests still hit a healthy step and a growing minority wait behind a full batch or a full KV pool. A platform can show a flat, healthy average while quietly failing its SLA for the slowest 1% during a rollout or a spike. That is exactly why the design targets 70% utilization (headroom against the cliff) and autoscales on queue depth (the thing that moves before the mean does). Common wrong answer to avoid: "Average latency is fine, it's within target." The mean hides the queue; by the time it moves, p99 has been broken for a while and the fleet is already saturating.

Deeper follow-ups

• How would you serve a model too large for one GPU — tensor parallelism (split each layer across GPUs) versus pipeline parallelism (split layers into stages) — and how does each change the batching and the failure blast radius?
• How would prefix caching change the math when 90% of requests share a 2,000-token system prompt, and what breaks if you offload cold prefixes to CPU RAM?
• How do you set max_batch adaptively as request length distribution shifts through the day, rather than fixing it?
• How would you support both latency-sensitive interactive traffic and throughput-oriented batch/offline traffic on the same fleet without the batch jobs starving the interactive ones?
• What changes for a classic low-latency model (a fraud classifier at p99 ≤ 10 ms, single forward pass) versus this autoregressive LLM — where does continuous batching stop being the answer?
• How would you run a shadow deployment (mirror real traffic to v43 without returning its output) to catch a regression before any user sees v43 at all?

How this round is scored

Interviewers use the inference platform to see whether you reason from the accelerator outward. The strong signal is recognizing early that the GPU is the scarce, expensive resource and that batching — not routing or storage — is where throughput and p99 are decided, then deriving the fleet size from the model's memory footprint and the KV-cache ceiling rather than guessing. Seniority shows up in the tradeoff discussions — batch size versus latency, cold start versus cost, shared versus dedicated isolation — where you move real numbers (batch 45 → TPOT 22 ms → ~100 GPUs) and name both sides. The rollout and failure sections separate people who have operated these systems from those who have only drawn them: canarying at 1% to cap blast radius, autoscaling on leading signals because cold start loses the reactive race, and refusing to replicate an ephemeral KV cache. Doing the memory and throughput math out loud, and using it to justify the fleet size and the batch cap rather than as decoration, is what pushes an answer from "correct" to "senior."