Skip to content

00. Design a Distributed Message Queue

~20 min read · Level: advanced · Part 1 of 4 (Overview → HLD → LLD → Q&A)

Problem

A distributed message queue accepts a torrent of events from many producers, stores them durably, and lets many independent consumers read them at their own pace. This is the product behind Apache Kafka, and behind the managed services built on it — Confluent Cloud, AWS MSK, Redpanda, Azure Event Hubs. A payments service publishes every transaction; a fraud detector, a ledger, a data-warehouse loader, and a real-time dashboard each read that same stream without knowing about one another. The queue's job is to sit between them so producers never wait on consumers and a slow reader never blocks a fast one.

What makes this a genuine system-design question, rather than "an array you append to," is the combination of three demands that pull against each other: enormous append throughput, strict ordering for related events, and the ability for a reader to fall far behind and still recover everything. A naive queue satisfies any one of these and breaks on the other two. The whole design is about holding all three at once.

To keep the reasoning concrete, thread one scenario through every file: a platform ingesting 1,000,000 events per second, where events carrying the same key (say, one account_id) must be delivered in the exact order they were produced, and where one consumer group — the data-warehouse loader — falls a full day behind during an outage and must catch up without losing a single event. That one number (1M/s), that one guarantee (per-key order), and that one recovery (a day-behind reader) will test every decision below.

Functional requirements

  • Publish: a producer sends an event to a named topic and gets an acknowledgement once it is durably stored.
  • Subscribe / consume: a consumer reads events from a topic starting at a position it controls, and can re-read from an earlier position.
  • Ordering: events sharing a key are delivered to a consumer in production order. Ordering across different keys is not promised.
  • Consumer groups: a group of consumers splits a topic's work so each event is processed by exactly one member of the group, and multiple independent groups each get the full stream.
  • Retention & replay: events are kept for a configured window (time or size), during which any consumer can rewind and reprocess.
  • Delivery semantics: at-least-once by default, with an exactly-once option for pipelines that cannot tolerate duplicates.

De-scoped for this round, and worth naming so the interviewer hears it as a choice: per-message priority queues, arbitrary per-message TTL and delayed delivery, server-side content filtering or routing (the RabbitMQ/SQS feature set), and cross-datacenter mirroring. These are real products in their own right, but bolting them on does not change the core log architecture, and some of them actively fight it.

Non-functional requirements

The dominant constraint is sustained sequential write throughput with durable ordering — everything downstream follows from the decision to model the system as an append-only log rather than a random-access datastore.

  • Throughput: sustain 1,000,000 events/second of writes (≈1 GB/s at 1 KB/event) and several times that in reads, since multiple consumer groups each read the full stream.
  • Durability: an acknowledged event must survive broker failures. Losing an acknowledged transaction is unacceptable; this forces replication before ack.
  • Ordering: total order within a partition, for every event that shares a routing key. This is the guarantee that shapes the data model.
  • Availability: producers must keep publishing through single-broker failures. A brief unavailability of one partition is tolerable; losing the whole cluster's write path is not. Target the write path staying up through the loss of any one broker with zero data loss.
  • Elastic reader lag: a consumer must be able to fall hours or a day behind and recover, which turns retention from a cleanup setting into a correctness parameter.

Scale estimation

Take the scenario's 1,000,000 events/second at an average 1 KB per event. That is 1 GB/second of ingress. Kafka keeps replication.factor = 3 copies of every partition for durability, so the cluster actually writes 3 GB/second across its disks and replication links. Per day the raw volume is 1,000,000 × 86,400 = 86.4 billion events, or 86.4 TB/day of raw data, and ~259 TB/day once replicated.

Retention is where the scenario bites. If the warehouse loader can fall a full day behind, the log must still hold at least a day of data when it comes back — and you never size retention at exactly the worst observed lag, because the segment the consumer is reaching gets deleted the instant it arrives. Pick 7 days of retention to leave six days of slack: that is 605 TB raw, or ~1.8 PB on disk after 3× replication. Retention stopped being a hygiene knob and became the parameter that decides whether a day-behind consumer recovers or hits OffsetOutOfRange and loses data.

Broker count falls out of the write rate. Budget a broker at roughly 150 MB/s of sustained replicated write once you leave headroom for reads, compaction, and replication traffic; 3 GB/s ÷ 150 MB/s ≈ 20 brokers as a floor, and you would run more for read fan-out and failure headroom. Partition count falls out of parallelism: pick 200 partitions for the topic, so each partition carries 1,000,000 ÷ 200 = 5,000 events/s ≈ 5 MB/s — comfortably under a single partition's ceiling, and it caps the consumer group at 200 parallel readers, which we will need for catch-up.

Now the catch-up math. A day behind is 86.4 billion events of backlog. Live traffic is still arriving at 1M/s, so a consumer group that merely matches 1M/s never gains ground. Run 200 consumers (one per partition) each processing ~30,000 events/s — a 6× overcapacity per partition — for 6,000,000 events/s of group throughput. That drains the backlog at 6M − 1M = 5M events/s of surplus, so 86.4 billion ÷ 5 million ≈ 17,300 seconds ≈ 4.8 hours to fully catch up while live traffic keeps flowing. The read amplification during catch-up is real: the group is pulling 6 GB/s off disk, which is why brokers must be sized for read bandwidth well above the 1 GB/s ingress.

API sketch

# Producer
producer.send(topic="txns", key=account_id, value=event, acks="all")
  -> RecordMetadata { partition, offset }        # offset is the event's position in the partition

# Consumer (pull model)
consumer.subscribe(topics=["txns"], group_id="warehouse-loader")
records = consumer.poll(timeout_ms=500)          # returns a batch across assigned partitions
consumer.commit(offsets={ (topic, partition): last_processed_offset + 1 })

# Positioning / replay
consumer.seek(partition, offset)                 # rewind or fast-forward within retention
consumer.seek_to_beginning(partition)            # replay from the oldest retained event

# Admin
admin.create_topic(name="txns", partitions=200, replication_factor=3)

Solutioning

Start from the throughput and durability numbers and the shape is forced. To take 1 GB/s of writes and never lose an acknowledged event, you want the write to be a sequential append to a replicated file, not an insert into an indexed structure — sequential disk writes on commodity hardware run an order of magnitude faster than random ones, and an append-only log has no update-in-place to make durable. So the core abstraction is a partitioned, append-only commit log: each topic is split into partitions, each partition is an ordered file living on a set of brokers, and each event lands at a monotonically increasing offset. The consumer's read position is just an integer offset it stores, which is what lets a reader move at its own speed, rewind, or fall a day behind and resume. The reframing that carries the whole design: a message queue is not a datastore you push into and pop out of; it is a log that many readers scan at their own offsets. Nothing is deleted when it is read; retention, not consumption, decides when data goes away.

The first defining tradeoff is per-partition ordering versus parallelism, and it is the one candidates get wrong. Kafka only guarantees order within a partition, and a partition is read by at most one consumer in a group at a time. So ordering and parallelism are the same dial: more partitions buys more parallel consumers but scatters order across them. The resolution is to make ordering a property of the key, not the topic. Route events by hashing their key to a partition (partition = hash(account_id) % 200), so every event for one account lands in one partition and is therefore totally ordered, while different accounts spread across all 200 partitions for parallelism. The memory hook: ordering in Kafka is not a global setting you request; it is a per-partition side effect you engineer through your choice of key. Get the key right and you get exactly as much ordering as you need and not one bit more, which is what lets the throughput scale.

The second tradeoff is at-least-once versus exactly-once, and it is really a question of where you pay. The default is at-least-once: a consumer processes a batch, then commits its offset, and if it crashes between processing and committing, it reprocesses that batch on restart — correct, cheap, but duplicates are possible, so downstream logic must be idempotent. Exactly-once is available through the idempotent producer (each producer stamps a sequence number so the broker deduplicates retries) plus transactions that atomically bind the messages written and the offsets committed, but it adds coordination that cuts effective throughput noticeably and only holds within Kafka's own boundary. The pragmatic call at 1M/s is at-least-once with idempotent consumers by default, reserving exactly-once for the pipelines — a financial ledger, say — where a duplicate is a correctness bug rather than a nuisance.

The third tradeoff is retention versus storage cost, and the scenario forces it into the open. Longer retention is what makes the day-behind warehouse loader recoverable, but at 259 TB/day replicated, every extra day of retention is another 259 TB of disk you buy and power. Seven days costs ~1.8 PB; thirty days would cost ~7.8 PB for a safety margin most systems never use. The resolution is to set retention from the worst tolerable consumer lag plus a wide margin, monitor real consumer lag continuously, and alert long before any group's lag approaches the retention edge — because the failure is silent and asymmetric: a consumer that crosses the retention boundary does not slow down, it loses the data that aged out from under it. The following files take the log, the offsets, the replication protocol, and these three tradeoffs down to components (HLD) and then to on-disk formats, the ISR replication algorithm, and the concurrency corners (LLD).