03. Rate Limiter — Interview Q&A

~15 min read · Part 4 of 4 (Overview → HLD → LLD → Q&A)

These are the questions an interviewer actually asks once the diagram is on the board. Each answer is the strong version, followed by the wrong answer that quietly sinks candidates.

Q1. You need to enforce 100 requests/second per API key across 50 stateless gateways. Why can't each gateway just count to 100 locally? Because no gateway sees all of a key's traffic — the load balancer spreads it across all 50 with no affinity — so if each enforced 100/s locally the fleet would allow 50 × 100 = 5,000/s, fifty times the limit. The count has to be shared. The clean framing is that this is not a counting problem, it's a coordination problem: the hard part isn't counting requests on one box, it's agreeing on one count across 50 without a round-trip per request. The working answer is a central counter sharded by API key so all 50 gateways read and write the same number for a given key, plus a local token bucket that absorbs obvious over-limit traffic so the shard isn't hit on every request. Common wrong answer to avoid: "Give each gateway a counter and it'll be fine." It's fine only if traffic is perfectly even and you divide the limit — and then uneven traffic either overshoots by the fleet factor or throttles legitimate users to a fraction of their limit.

Q2. Central counter or local counters — how do you actually resolve accuracy versus latency? Neither pole works alone. A pure central counter is accurate to one increment because all gateways share it, but it puts a ~0.4 ms round-trip and one store op on every request — 500k ops/s at peak. Pure local is zero-latency but wrong by the fleet factor. Resolve it by making the count central and authoritative but sharded by API key so no single node is the bottleneck (500k ops/s over 8 shards is ~62k each), then add a local token bucket that short-circuits blatantly over-limit keys without a round-trip. Accuracy comes from the shared shard; latency and bottleneck-avoidance come from sharding plus the local fast-path. Common wrong answer to avoid: "Route everything through one Redis for correctness." That's accurate and a single bottleneck — you've moved the problem, not solved it, and one hot key or one node failure takes the whole limiter down.

Q3. Token bucket or sliding window — which algorithm, and why? They answer different questions. A token bucket (capacity + refill rate) naturally allows bursts up to the bucket size and is the right model when you want to permit short spikes — 100/s steady but tolerate a burst of 150. A fixed-window counter is one integer but permits a 2× boundary burst: 100 requests at t=0.99s and 100 at t=1.01s is 200 in 20 ms while each window reads "100, legal." A sliding-window log is exact but stores every timestamp. The default here is the sliding-window-counter — two counts blended by how far into the window the clock has moved — because it kills the boundary burst, costs ~40 bytes/key instead of the log's ~800 bytes/key, and its error is a fraction of a percent. Pick token bucket when burst shaping is the goal; pick sliding-window-counter when smooth per-second enforcement is. Common wrong answer to avoid: "Fixed window, it's simplest." It is, until a client learns to fire at the window boundary and legally does double your limit in the seam.

Q4. Walk the sliding-window-counter math for the 100/s key. Windows are 1 second. Say the previous window counted 82 and the current window has 55 so far, and the clock is 550 ms into the current window (elapsed = 0.55). The estimated trailing-second rate is prev × (1 − elapsed) + cur = 82 × 0.45 + 55 = 36.9 + 55 = 91.9, under 100, so allow. A few requests later cur = 64, elapsed = 0.55: 82 × 0.45 + 64 = 100.9 ≥ 100, so deny with a Retry-After set to when the previous window's weight decays enough to free a slot. A fixed window would have seen cur = 64 < 100 and wrongly allowed, blind to the 82 that arrived in the last 450 ms. The approximation assumes the previous 82 were evenly spread; that's the only error, and it's tiny. Common wrong answer to avoid: "Just count requests in the current second." That's the fixed window, and it's the boundary-burst bug — it can pass 82 + 64 = 146 in one trailing second while calling every window legal.

Q5. How do you keep two gateways from both allowing the 100th request? Make the counter's read-compute-write atomic on the shard. If you GET the count, compare in the gateway, and SET, two gateways both read 99, both decide "under 100," both write 100, and two requests took the last slot. Instead run a Lua script on the Redis node that reads the counters, computes the rate, and increments-or-rejects as one indivisible operation, so the second caller sees the first's write. The store, not application code, arbitrates the race. Common wrong answer to avoid: "Read the counter, check it, then increment." That read-check-increment is a lost-update race; under load it leaks requests past the limit exactly when the limit matters most.

Q6. Redis is unreachable for a key's shard — fail open or fail closed? This is a policy choice about who to hurt, and the limiter must decide in under a millisecond rather than wait. Fail-open lets requests through uncounted, protecting your users but briefly exposing the backend to overload; fail-closed rejects them, protecting the backend but 429-ing legitimate traffic — a self-inflicted outage. The default here is fail-open with the local token bucket as a ceiling: with the shard down, each of the 50 gateways falls back to enforcing 100/50 = 2/s locally, so the fleet still caps the key near its 100/s intended rate instead of flooding or blacking out. When the shard returns, gateways resume the shared count and the local buckets drain. Common wrong answer to avoid: "Fail closed to be safe" (or "fail open, requests just pass"). Naive fail-closed turns a cache blip into an API outage; naive fail-open removes the limit entirely. The bounded local-ceiling fallback is what makes either safe.

Q7. One API key is doing 100k/s of abuse. How does the system survive it, and where does the load land? Recognize it's a hot-key problem on the write path: all of that key's counter traffic hashes to one shard. The local token bucket on each gateway is the defense — once a gateway has seen the key blow past its limit, it denies locally without a round-trip, so 100k/s of abuse is absorbed across the 50 gateways' memory and the shard only ever sees roughly the allowed rate (~100/s) plus a trickle of probes. The abusive flood costs nothing downstream and nearly nothing on the shard. For a legitimate very-high-limit key that's genuinely hot, split its counter across N shards each enforcing limit/N to spread the op load. Common wrong answer to avoid: "Add more Redis shards." Sharding spreads distinct keys; one hot key still lands on one shard. Only the local fast-path takes that flood off the network.

Q8. How much does this cost in memory and store ops at your scale? At 1M active keys and 500k req/s peak: store ops are up to 500k/s, which is why the counter is sharded — 8 shards at ~62k ops/s each, well inside a single node's ~100k–150k envelope. Memory depends on the algorithm and that's a design lever: the sliding-window-counter is ~40 bytes/key, so 1M × 40 B = 40 MB (≈5 MB/shard); a sliding-window-log would be up to 100 timestamps/key at 100/s, 1M × 800 B = 800 MB. That 40 MB vs 800 MB, 20×, is the memory-versus-precision tradeoff, and the extra precision buys accuracy the platform doesn't need. Counters TTL after two windows, so idle keys cost nothing. Common wrong answer to avoid: "Memory doesn't matter, Redis has plenty." It matters because the algorithm that costs 20× more memory (the log) gives you exactness you're not billing on — the cheap approximation is the right call, and knowing why is the signal.

Q9. Would you route each API key to a fixed gateway so it can count locally with no round-trip? It's tempting and it does buy exact local counting with zero round-trips — the LB consistently-hashes each key to one gateway that holds its counter. The costs are real: the LB is no longer a dumb spreader, a gateway failure reshuffles keys and briefly double-counts during the handoff, and a hot key overloads its one home gateway with no way to shed. The mainstream choice keeps gateways interchangeable and the count in a sharded store, trading one round-trip (mitigated by the local fast-path) for a stateless tier and a hot key that's absorbed in memory rather than pinned to one box. Common wrong answer to avoid: "Affinity routing removes the round-trip, so it's strictly better." It moves the hot-key and failover problems onto the LB and the home gateway — you've traded a solved problem for two harder ones.

Q10. Why not just increment a counter on every request and reset it every second? That's the fixed-window counter, and it has two flaws worth naming. First, the boundary burst: it allows up to 2× the limit across two adjacent windows. Second, if the reset is a client-side SET 0 rather than a natural TTL, a crash between increment and reset can leave a stale count that locks the key out or a missing TTL that never expires. The sliding-window-counter fixes the burst with a two-count blend, and doing the increment and EXPIRE in one atomic Lua block fixes the reset hazard — the key expires itself after two windows with no reset command to fumble. Common wrong answer to avoid: "Increment and set a one-second expiry, done." The separate INCR and EXPIRE can be split by a crash, leaking a key with no TTL, and you still have the boundary burst.

Q11. A client keeps getting 429s and retrying immediately. What should the system do, and whose bug is it? The limiter counts arrivals, so an immediate retry is a new request that legitimately consumes a slot and makes the client's own throttling worse — it's the client's bug, and the fix is protocol, not counting. Return Retry-After and X-RateLimit-Reset so well-behaved clients back off until the window frees up, and consider a secondary limiter that penalizes clients ignoring 429s (e.g. a longer cool-down for a key generating sustained rejections). The limiter should not try to dedupe retries as if they were idempotent operations — it's rate-limiting arrivals, not logical actions. Common wrong answer to avoid: "Dedupe the retries so they don't count." Retries are real load hitting your backend; not counting them defeats the purpose, and the limiter can't tell a retry from a fresh request anyway.

Q12. How do you handle different limits per plan, per route, and per key at once? Model each as a descriptor and evaluate the applicable limits together, letting the tightest one govern — a free-plan key on an expensive route is bound by whichever of {plan limit, route limit, key limit} is smallest. Limits live in a policy store as data that gateways watch and hot-reload, so raising a paying customer's limit or clamping an abuser is a config edit that takes effect in seconds, not a redeploy. Each descriptor gets its own counter key (rl:{key}:{route}), checked in the same atomic path. Common wrong answer to avoid: "One global limit per key." Real APIs need per-route and per-plan limits, and hard-coding them means every limit change is a deploy — slow exactly when you need to react to abuse fast.

Deeper follow-ups

• How would you bound the overshoot of a fully-distributed (no per-request round-trip) design where each gateway syncs its local count to the shard every 100 ms — what's the worst-case burst, and how does sync interval trade against store load?
• How would you implement a token bucket entirely inside Redis with a Lua script (or CL.THROTTLE), and what does it store versus the sliding-window-counter?
• How do you keep the 50 gateways from disagreeing on window boundaries under clock skew, and why does computing time inside the Lua script fix it?
• How would you rate-limit by a cost other than one-per-request — e.g. a request that weighs 10 tokens because it's an expensive query?
• How would you extend this to a global, multi-region API where a key's traffic hits data centers on two continents, and what consistency would you accept on the shared count?
• What changes if you need the limiter to be exact for billing (metering) rather than approximate for protection?

How this round is scored

Interviewers use the rate limiter to see whether you understand that a limiter is a distributed-systems problem, not a counting exercise. The strong signal is naming early that 50 local counters allow 5,000/s and driving the whole design from the accuracy-versus-latency tension that follows — landing on central-but-sharded with a local fast-path, rather than either a single-Redis bottleneck or naive per-server counters. Seniority shows in the tradeoff discussions: token bucket versus sliding window with the memory numbers, fixed-window boundary burst, and the fail-open-versus-fail-closed call with a bounded local fallback rather than a hand-wave. The failure-mode thinking — hot keys absorbed locally, atomic Lua for the lost-update race, EXPIRE in the same block, clock skew handled in the store — separates candidates who have operated a limiter from those who have only drawn one. Doing the ops-per-shard and memory-per-algorithm math out loud, and using it to justify the approximation rather than as decoration, is what pushes an answer from "correct" to "senior."