02. WhatsApp / Messenger — Low-Level Design¶

~18 min read · Part 3 of 4 (Overview → HLD → LLD → Q&A)

Data models¶

CREATE TABLE message (
    conv_id       BIGINT NOT NULL,
    seq           BIGINT NOT NULL,        -- monotonic per conversation → ordering
    server_msg_id BIGINT NOT NULL,        -- global unique
    sender_id     BIGINT NOT NULL,
    client_msg_id UUID NOT NULL,          -- sender-generated → dedup
    body          BYTEA,                  -- ciphertext for E2E
    created_at    TIMESTAMP NOT NULL,
    PRIMARY KEY (conv_id, seq)
);                                         -- sharded by conv_id
CREATE UNIQUE INDEX ux_dedup ON message (conv_id, client_msg_id);

-- Inbox queue (per user):  inbox:{user_id} = ordered (conv_id, seq) entries undelivered
-- Session registry (Redis): sess:{user_id} = { gateway_node, conn_id, expires_at }
-- Receipts:  receipt(conv_id, seq, user_id, state, ts)

Two keys carry the guarantees. PRIMARY KEY (conv_id, seq) makes ordering a property of storage — reading a conversation in seq order is the correct order. The unique index on (conv_id, client_msg_id) makes dedup an insert conflict: a retransmitted message fails the unique constraint, so it's stored once no matter how many times Alice's client retries.

Component internals¶

Message service — persist-first, dedup, sequence¶

def handle_send(sender, msg):
    seq = sequencer.next(msg.conv_id)                  # monotonic per conversation
    sid = snowflake()
    try:
        store.insert(msg.conv_id, seq, sid, sender.id, msg.client_msg_id, msg.body)
    except UniqueViolation:                            # dedup: already stored
        existing = store.get_by_client_id(msg.conv_id, msg.client_msg_id)
        return Ack(existing.server_msg_id, "sent")     # idempotent ack
    ack(sender, Ack(sid, "sent"))                      # durable BEFORE delivery
    deliver_or_enqueue(msg.conv_id, seq, sid)
    return

The ordering is deliberate: persist and ack before attempting delivery. If delivery fails, the message is safe and will redeliver; if the ack is lost, the retransmit dedups. Alice never loses a message and never sees it twice.

Delivery router — online vs offline¶

def deliver_or_enqueue(conv_id, seq, sid):
    for recipient in members(conv_id) - {sender}:
        sess = session_registry.get(recipient)
        if sess and gateway_alive(sess.node):
            forward(sess.node, sess.conn_id, DELIVER(sid, ...))   # push now
        else:
            inbox.enqueue(recipient, conv_id, seq)                # offline → queue

For Bob offline, there's no live session, so the message enqueues. This is the same code path whether Bob is offline, mid-reconnect, or on a dead node — persist-first means enqueue is always a safe fallback.

Reconnect drain — ordered catch-up¶

def on_reconnect(user, last_seq_by_conv):
    session_registry.put(user, this_node, conn_id, ttl=30s)
    for conv_id, last_seq in last_seq_by_conv.items():
        for m in store.range(conv_id, after=last_seq):   # in seq order
            push(user, DELIVER(m.server_msg_id, ...))
    inbox.clear_delivered(user)

Bob sends the last seq he has per conversation; the server streams everything after it in order. Alice's "running late" arrives in its correct sequence slot even though it was sent while Bob was offline.

Core algorithm — per-conversation ordering under concurrency¶

sequencer.next(conv_id):
    # sequence lives on the conv_id's shard, so one writer per conversation
    return atomic_increment(seq_counter[conv_id])

client render:
    buffer messages, sort by seq within conv_id, render contiguously
    if a gap in seq (missing seq N): request retransmit of N before rendering past it

Ordering is per conversation, not global — two different conversations need no coordination. Because a conversation is pinned to one shard, its sequence is a simple atomic counter; the client detects gaps (a missing seq) and requests the missing message before rendering later ones, so order is never violated even if packets arrive scrambled.

Sequence diagram — offline delivery + receipts¶

Alice     A-gateway   Msg svc     Store     Bob-inbox    Bob        B-gateway
 │ SEND     │            │           │          │         (offline)     │
 ├──────────▶│ persist ──▶│ insert ──▶│          │           │           │
 │◀── ACK sent ──────────┤           │          │           │           │
 │           │ Bob offline?│ yes ─────┼─ enqueue▶│           │           │
 │           │            │           │          │  reconnect (last_seq) │
 │           │            │◀──────────┼──────────┼───────────┼───────────┤
 │           │            │ stream after last_seq (in order) ▶│ store+ACK │
 │◀─ RECEIPT delivered ───┤◀──────────┼──────────┼───────────┤           │

Concurrency and edge cases¶

Retransmit storm: the unique (conv_id, client_msg_id) index makes every duplicate a no-op insert; the server re-acks idempotently.
Two devices, same user: each device is its own session; the message fans out to all of a user's sessions, deduped per device by server_msg_id.
Sequence gap on client: client blocks rendering past the gap and requests the missing seq, guaranteeing contiguous order.
Presence flapping: presence is TTL heartbeats; a missed heartbeat marks last-seen rather than hard-offline, avoiding flicker.
Group ordering: each member reads the shared conversation sequence, so all members see the same order; per-member delivery state (delivered/read) is tracked separately.
Message expiry: undelivered messages for a permanently-gone device expire after a bound (e.g. 30 days) to cap inbox growth.
E2E encryption: the server stores/forwards ciphertext (body BYTEA) and never sees plaintext; ordering, dedup, and receipts all work on metadata, not content.