03. Distributed Message Queue — Interview Q&A

~15 min read · Part 4 of 4 (Overview → HLD → LLD → Q&A)

These are the questions an interviewer asks once the log and its partitions are on the board. Each answer is the strong version, followed by the wrong answer that quietly sinks candidates.

Q1. How does the queue guarantee ordering, and what exactly is ordered? Ordering is guaranteed only within a partition, and a partition is chosen by hashing the event's key. So all events for one account_id land in one partition, get monotonically increasing offsets, and are read in that order by the single consumer that owns the partition in a group. There is no ordering across different keys, and asking for it would mean one partition and no parallelism. The design move is to key by the entity whose order matters — the ordering you need becomes a side effect of the key, and everything else spreads across all 200 partitions for throughput. Common wrong answer to avoid: "Kafka delivers messages in the order they were sent." Not globally — only per partition. Claiming total order across a topic reveals you have not connected ordering to the partition/key model.

Q2. Why an append-only log instead of a database-backed queue? Because the workload is 1 GB/s of writes that must be durable and ordered, and sequential appends to a replicated file are roughly an order of magnitude faster than random inserts into an indexed structure — there is no B-tree to rebalance, no update-in-place to make durable, and reads are sequential scans served by zero-copy from page cache. Consumption does not delete anything; a consumer's position is just an offset it stores, which is what lets many groups read the same stream independently and lets any of them rewind. A traditional queue that deletes on dequeue cannot give you replay or multiple independent readers. Common wrong answer to avoid: "Use a database table with a status column and poll for unprocessed rows." That reintroduces random IO, row locking, and a delete/update per message — it collapses well below 1M/s and gives no replay.

Q3. A consumer group falls a full day behind during an outage — how does it catch up without losing data? First, this is a retention problem before it is a speed problem: the day-old offsets must still point at data on disk. With 7-day retention against 1-day lag, every committed offset resolves, so the group resumes exactly where it stopped. Then it is a throughput problem: it is 86.4 billion events behind while 1M/s keeps arriving, so it must out-run production. Running 200 consumers at ~30,000 events/s each gives 6M/s, a 5M/s surplus over live traffic, draining the backlog in about 4.8 hours — and because each account stays on one partition read by one consumer, catch-up replays each account's events in original order. Had retention been 24 hours to save disk, the segment under its offset would have aged out and it would have silently skipped a day. Common wrong answer to avoid: "Just add more consumers and it'll catch up." Only if retention still holds the data — and only up to one consumer per partition (200 here); a 201st consumer sits idle, and no consumer count recovers data that already aged out.

Q4. At-least-once, at-most-once, or exactly-once — how do you choose and implement each? It comes down to the order of process and commit. Commit after processing is at-least-once: a crash before the commit reprocesses the batch, so duplicates are possible and downstream logic must be idempotent — the sane default. Commit before processing is at-most-once: a crash after the commit skips the batch, losing data, which you almost never want. True exactly-once needs the idempotent producer (sequence numbers dedup retries) plus transactions that atomically bind the messages written and the offsets committed, with consumers reading read_committed. Exactly-once is correct but cuts effective throughput from the coordination, so reserve it for pipelines like a ledger where a duplicate is a correctness bug, and use at-least-once with idempotent consumers everywhere else. Common wrong answer to avoid: "Turn on exactly-once for everything so we never think about duplicates." It only holds within Kafka's boundary, it costs throughput at 1M/s, and it does not save you from duplicates your external side-effects create.

Q5. How is a write kept durable, and what does acks=all actually wait for? Each partition is replicated to 3 brokers; the subset caught up to the leader is the in-sync replica set (ISR). With acks=all the leader releases the producer's ack only once every ISR member has the batch — mechanically, once the high-water mark, the minimum log-end offset across the ISR, advances past it. Since only an ISR member can be elected leader, any single broker can then die without losing an acknowledged event. Setting min.insync.replicas=2 means if the ISR shrinks to one replica, the partition refuses acks=all writes rather than acknowledge data that lives on a single disk. Common wrong answer to avoid: "acks=all waits for all replicas." It waits for all in-sync replicas; a follower that fell behind is dropped from the ISR precisely so one slow disk cannot block every producer.

Q6. What sizes the cluster — how many brokers and partitions for 1M events/s? At 1 KB/event that is 1 GB/s ingress, tripled to 3 GB/s by RF=3. Budgeting ~150 MB/s of sustained replicated write per broker puts the floor near 20 brokers, and you run more for read fan-out. Partition count is set by parallelism, not just throughput: 200 partitions puts each at 5,000 events/s (~5 MB/s), well under a partition's ceiling, and caps a consumer group at 200 parallel readers — which is exactly the parallelism the day-behind catch-up needs. Reads often size the cluster more than writes: three live groups plus one catching up is ~9 GB/s of reads against 1 GB/s of writes. Common wrong answer to avoid: "Use a few big partitions to keep it simple." Partition count is your maximum consumer parallelism; under-partitioning caps a group's throughput and makes it impossible to catch up when it falls behind.

Q7. How much storage does retention cost, and how do you set it? 86.4 billion events/day × 1 KB is 86.4 TB/day raw, ~259 TB/day after 3× replication. Seven days is ~1.8 PB on disk; thirty days would be ~7.8 PB. You set retention from the worst tolerable consumer lag plus a wide margin — not the disk you would like to save — because the failure is silent: a consumer that crosses the retention edge does not slow down, it loses the data that aged out. Then you monitor real per-group lag continuously and alert long before any group nears the edge. Tiered storage (cold segments to object storage) is the lever to keep long retention without paying for local SSD across the whole window. Common wrong answer to avoid: "Set retention to a day to save money since consumers are usually caught up." That is exactly the config that turns a one-day consumer outage into permanent data loss.

Q8. What happens when a broker leading a partition dies? The controller detects the failure and elects a new leader from the ISR — a replica guaranteed to hold every acknowledged event, since the high-water mark never advanced past data an ISR member lacked. Producers refresh metadata and resend to the new leader; that partition's writes pause for the election window (seconds) and resume with zero data loss. Consumers likewise re-fetch from the new leader at their existing offsets. The only lossy scenario is unclean leader election — promoting an out-of-sync replica when the whole ISR is gone — which you disable for a payments stream, accepting a stalled partition over lost transactions. Common wrong answer to avoid: "A load balancer routes around the dead broker." There is no stateless failover here; a specific partition has a specific leader, and recovery is leader election from replicas that hold the data, not rerouting.

Q9. One account_id suddenly produces 100,000 events/s — what breaks and what do you do? Hashing pins that key to a single partition, so one partition runs at 100k/s while its 199 siblings idle at 5k/s, and because a partition has one leader and one consumer-per-group, neither adding brokers nor adding consumers relieves it — the hot partition is the ceiling. The fix lives at the key: sub-key the hot account (account_id#shard) to spread its events across several partitions, which restores parallelism but gives up strict cross-shard ordering for that one account. That is the ordering-versus-parallelism dial again, applied surgically to the offending key rather than the whole topic. Common wrong answer to avoid: "Add more partitions or brokers." Extra capacity does nothing for a single hot key confined to one partition; only changing the key distribution helps.

Q10. Why is broker CPU low while the cluster is clearly working hard, and what should you watch instead? Because a broker's work is IO and page-cache movement, not computation — appends go to page cache, reads leave via zero-copy sendfile without deserializing, so a cluster can saturate its disks and NICs with CPU near idle. Watching CPU hides the real state. The signals that matter are per-group consumer lag (is anyone losing the race with production?), ISR shrink (are acks=all writes about to block?), and disk/network throughput. During an incident, the first graph to open is per-group lag over time: an upward-bending line means a group is falling behind, and its slope against the retention window tells you how many hours until it starts losing data. Common wrong answer to avoid: "CPU is low, so the cluster is healthy and has headroom." CPU is the wrong gauge for a log system; it can be melting its disks with CPU asleep.

Q11. A consumer hits one message it cannot process — what happens to everything behind it? Nothing behind it in that partition gets processed, because the consumer cannot advance its offset past the poison message, and the partition is strictly ordered — so one bad event stalls every event behind it for that key. The fix is a dead-letter topic: the consumer routes the un-processable event aside, commits past it, and keeps moving. That trades strict ordering on that one key for liveness on all the others, which is usually the right call, but you make it deliberately because it is a real ordering concession. Common wrong answer to avoid: "Skip the message and move on." Skipping without capturing it to a dead-letter topic silently drops data; the point is to preserve it and unblock the partition.

Q12. How do offsets and consumer groups let multiple teams read the same stream? Each consumer group stores its own committed offsets (in the internal __consumer_offsets topic), so the fraud detector, the ledger, and the warehouse loader each read the full stream at an entirely independent position — one being a day behind does not affect the others, because reading does not consume or delete anything. Within a group, the coordinator assigns each partition to one member so each event is handled once per group; add a brand-new group and it can replay from the earliest retained offset without touching producers or existing groups. Offsets-as-consumer-state is what turns one log into a shared source of truth for many independent pipelines. Common wrong answer to avoid: "Once a message is read it's gone, so you need to copy the stream per team." That is a delete-on-read queue model; here consumption is a cursor, not a pop, and every group reads the same retained log.

Deeper follow-ups

• How would you migrate a topic from 200 to 400 partitions without breaking per-key ordering for keys already in flight?
• How does log compaction differ from time/size retention, and when would you compact instead of expire (e.g. a changelog of latest state per key)?
• Walk through exactly-once for a read-process-write pipeline: what is in the transaction, and what does a read_committed consumer see during an in-flight transaction?
• How do KRaft and ZooKeeper differ as the metadata quorum, and why did the project move off ZooKeeper for large clusters?
• How would tiered storage change your retention math and your catch-up latency for the day-behind consumer?
• How do you prevent rebalance storms when consumers restart frequently (static membership, cooperative rebalancing), and what do they cost?

How this round is scored

Interviewers use the message queue to see whether you reason from the log abstraction rather than reaching for a database with a queue bolted on. The strong signal is naming ordering as a per-partition, per-key property early and deriving parallelism, hot keys, and consumer groups from it, instead of promising global order you cannot deliver. Seniority shows in the tradeoff discussions — per-partition order versus parallelism, at-least-once versus exactly-once, retention versus disk — where you name both sides and put numbers on them. The retention-and-catch-up scenario is the discriminator: candidates who have operated Kafka immediately treat a day-behind consumer as a retention-configuration question and know that consumer lag against the retention window is the graph that matters, while candidates who have only read about it try to solve a data-loss problem with more consumers. Doing the throughput and storage math out loud — 1 GB/s to 3 GB/s replicated, 259 TB/day, ~4.8 hours to drain 86.4 billion events — and using it to justify partition count and retention rather than as decoration is what pushes an answer from correct to senior.