Skip to content

02. Multi-Channel Notification System — Low-Level Design

~20 min read · Part 3 of 4 (Overview → HLD → LLD → Q&A)

The HLD named the boxes. This file opens the four that carry the design's weight — fanout, the preference + template pair, retry/dedup, and rate control — and pins down the data, the algorithms, and the concurrency corners where a notification system actually breaks: double-sends, lost fanout, and a rate spiral.

Data models

The notification (campaign) record is the system of record for "what was requested." It tracks lifecycle and holds rolling counts, not the 14M individual messages.

CREATE TABLE notification (
    id              UUID PRIMARY KEY,
    idempotency_key VARCHAR(128) UNIQUE,       -- dedups the SUBMIT (client retry of POST)
    template_id     VARCHAR(64)  NOT NULL,
    template_ver    INT          NOT NULL,     -- pinned at submit → in-flight edits can't change it
    audience_ref    JSONB        NOT NULL,     -- segment id or user-id batch pointer
    channels        TEXT[]       NOT NULL,     -- allowed set; per-user prefs filter within
    priority        SMALLINT     NOT NULL,     -- 0=bulk, 1=default, 2=high
    status          VARCHAR(16)  NOT NULL,     -- fanning_out | delivering | complete
    fanout_cursor   VARCHAR(64)  NULL,         -- checkpoint for resumable expansion
    created_at      TIMESTAMPTZ  NOT NULL
);

Two choices matter. idempotency_key is UNIQUE so a retried POST collapses to one notification and never triggers a second fanout. template_ver is pinned at submit time — the fanout renders against that exact version, so an editor changing the template mid-alert cannot alter messages already queued.

Preferences are keyed by user for a point read during fanout:

KEY  pref:{user_id}  →  {
    "channels":     { "push": true, "email": true, "sms": false },
    "categories":   { "breaking_news": {"sms": true}, "marketing": {"push": false} },
    "quiet_hours":  { "tz": "Asia/Kolkata", "from": "22:00", "to": "07:00" },
    "freq_cap":     { "marketing": {"n": 3, "per": "day"} },
    "override_urgent": true          -- allow breaking_news to pierce quiet hours
}

Category-level overrides sit above channel-level opt-ins: a user can mute SMS globally but keep it for breaking_news. Resolution is category-first, then the global channel flag as the default — the fanout must apply them in that order or it will over- or under-send.

The dedup and rate-limit state lives in the in-memory store as plain keys, no table:

dedup:{notification_id}:{user_id}:{channel}   →  "sent"        TTL 72h
rate:provider:{provider_id}                    →  token bucket  (refill 2000/s)
rate:user:{user_id}:{category}                 →  sliding count TTL = cap window

The dedup key deliberately excludes the provider: failover from primary to secondary must not mint a new identity, or the duplicate slips through. Delivery events are emitted to the columnar store with shape {notification_id, user_id, channel, provider, state, code, ts} and never stored in the operational database.

Component internals

Component 1 — Fanout Service (audience → per-channel messages)

Responsibility: expand an audience of up to tens of millions into individual, preference-filtered, template-rendered, deduplicable messages — in parallel, and resumably after a crash.

class FanoutWorker:
    def run(job: FanoutJob) -> None
    def _page_audience(ref, cursor) -> (list[user_id], next_cursor)
    def _expand_user(user_id, notif) -> list[Message]     # applies prefs + template

class Message:
    notification_id: UUID
    user_id:         int
    channel:         str          # push | email | sms
    idempotency_key: str          # f"{notification_id}:{user_id}:{channel}"
    rendered:        dict          # provider-ready payload
    priority:        int

The audience is partitioned across workers by a hash of user_id into slices; each worker owns a slice and pages through it, committing the fanout_cursor to the notification row after each page. _expand_user fetches the preference blob, computes the surviving channels for this notification's category, renders each via the Template Service, and returns one Message per channel. Emitting is a batched produce to the bus. Because the cursor is checkpointed per page, a crash re-runs at most one page's worth of already-emitted messages — harmless, since the dedup key drops them downstream.

Component 2 — Preference + Template resolution

Responsibility: decide whether to send on each channel, then produce the exact bytes to send.

Preference resolution is a pure function over the preference blob and the clock:

def resolve_channels(pref, notif) -> list[str]:
    out = []
    for ch in notif.channels:                       # allowed set from the request
        cat = pref.categories.get(notif.category, {})
        enabled = cat.get(ch, pref.channels.get(ch, False))   # category overrides global
        if not enabled:
            continue
        if in_quiet_hours(pref, now()) and not (notif.urgent and pref.override_urgent):
            continue
        if over_frequency_cap(pref, notif.category, user_id):
            continue
        out.append(ch)
    return out

Template rendering picks the (channel, locale) variant of the pinned template version and substitutes notif.data, enforcing channel limits — truncating a push body to the device limit, splitting or rejecting an SMS over 160 chars, assembling subject + HTML for email. Rendering is done at fanout time (not at send time) so the expensive template lookup happens once per message and the channel workers stay thin.

Component 3 — Channel worker (dedup, rate-limit, send, retry)

Responsibility: deliver a message at-least-once, never visibly twice, without exceeding provider limits.

def handle(msg: Message):
    # 1. Dedup gate: claim the key atomically. If already claimed, this is a duplicate.
    if not dedup.set_nx(msg.idempotency_key, "sending", ttl=72*3600):
        return ack(msg)                          # someone already (is) sent this; drop

    # 2. Rate gate: block until a provider token AND a per-user token are available.
    rate_limiter.acquire(provider_for(msg.channel), msg.user_id, msg.priority)

    # 3. Send via primary; fail over to secondary on provider-down.
    provider = pick_healthy_provider(msg.channel)
    resp = provider.send(msg.rendered)

    # 4. Interpret.
    if resp.ok:
        dedup.set(msg.idempotency_key, "sent", ttl=72*3600)   # confirm the claim
        record(msg, state="sent", provider=provider.id)
        return ack(msg)
    elif resp.retryable:                          # 429, 5xx, timeout
        dedup.delete(msg.idempotency_key)         # release claim so retry can proceed
        schedule_retry(msg, backoff(msg.attempt))
        return nack(msg)
    else:                                         # permanent: bad token, hard bounce
        dedup.set(msg.idempotency_key, "failed", ttl=72*3600)
        writeback_preference(msg)                 # e.g. mark device token dead
        route_to_dlq(msg)
        return ack(msg)

The set_nx in step 1 is the crux. It atomically claims the key, so two workers processing the same message (a bus redelivery, a fanout replay) race and exactly one wins; the loser drops. The claim is provisionally "sending" and only promoted to "sent" after the provider confirms — and critically, a retryable failure releases the claim (step 4b) so the legitimate retry is not itself dropped as a duplicate. The subtle hazard lives in that release: see the concurrency section for the timeout case where the send may actually have succeeded.

Core algorithm — fanout with dedup, walked through the 10M alert

Step through the breaking-news alert to see the numbers move.

  1. Submit. Ingestion writes one notification row (priority=2, channels=[push,email,sms], template_ver pinned), enqueues a fanout job over segment alerts_optin (10,000,000 users), returns 202. One durable write; the caller is done in milliseconds.

  2. Partition. The fanout job splits the 10M audience into 100 slices of 100,000 users by hash(user_id) % 100, one per fanout worker.

  3. Expand. Each worker pages its slice 1,000 users at a time (100 pages). Per user it reads the preference blob and runs resolve_channels. Across the audience the opt-in rates yield 90% push, 40% email, 10% SMS, so each 100k slice produces ~90k push + ~40k email + ~10k SMS = ~140k messages, and all 100 workers produce ~14,000,000 messages total. Each worker commits its cursor every page, so a crash costs at most 1,000 users of rework.

  4. Emit. Messages are produced to push.high, email.high, sms.high. At a few thousand emits/sec per worker × 100 workers, the entire 14M expansion lands on the bus in under a minute — fanout is not the bottleneck.

  5. Drain, per channel, paced by the rate limiter. Push workers pull 9M and, bounded by APNs throughput not the limiter, clear ~30,000/s → ~5 minutes. Email clears 4M near its provisioned rate → ~5 minutes. SMS is capped at the gateway's 2,000/s token-bucket, so 1M SMS drains in 1,000,000 / 2,000 = 500 s ≈ 8.3 minutes — the tail the overview predicted, and the reason SMS rides its own partition so a fraud OTP submitted at minute 4 is not stuck behind 600k undelivered blast messages.

  6. Dedup on every send. Each of the 14M handle calls does a set_nx. When a push worker crashes and the bus redelivers its last batch, those messages hit set_nx again, find the key claimed, and drop — so the redelivery adds zero duplicate pushes. This is where at-least-once stays invisibly single-delivery.

  7. Confirm via feedback. Over the following minutes provider webhooks promote sent → delivered; the notification's rolling counts converge toward delivered ≈ 13.9M with a residual failed (dead device tokens, hard email bounces) writing back to preferences.

Sequence diagram — a send with a bus redelivery (dedup in action)

Bus          WorkerA        WorkerB        Dedup(Redis)     Provider
 │  msg X ──▶ │               │                │              │
 │            ├─ set_nx(X) ───┼───────────────▶│ OK (claim A) │
 │            ├─ acquire token│                │              │
 │            ├─ send(X) ──────┼───────────────┼─────────────▶│  (slow...)
 │  (A's lease expires; bus redelivers X)      │              │
 │  msg X ─────────────────▶  │                │              │
 │            │               ├─ set_nx(X) ───▶│ FAIL (held)  │
 │            │               ├─ drop + ack ◀──┤              │
 │            │◀── 200 OK ─────┼────────────────┼──────────────┤
 │            ├─ set(X,"sent")─┼───────────────▶│ confirmed    │
 │◀── ack ────┤               │                │              │

WorkerA claims the key and sends; the bus redelivers X to WorkerB (A's processing outran its lease); B's set_nx fails because A holds the claim, so B drops the duplicate and acks. Exactly one message reaches the provider.

Concurrency and edge cases

  • Two workers, one message (bus redelivery). Resolved by the atomic set_nx claim: the first worker wins, the second drops. This is the common case the sequence diagram shows, and it is why the dedup gate is the first thing a worker does, before spending a rate-limit token or a provider call.

  • The timeout ambiguity (at-least-once's hard corner). A worker sends, the provider actually delivers, but the ack times out. The worker sees a retryable error and releases the claim, so the retry re-sends — a genuine duplicate the dedup key cannot catch, because from the store's view the first attempt never confirmed. This is the irreducible cost of an external provider: you choose the lesser evil per channel. For SMS/email, prefer a slightly longer claim hold and treat ambiguous timeouts as sent (risk a missed delivery over a paid duplicate); for push, release and retry (risk a rare double-buzz over a missed alert). There is no exactly-once across a boundary you cannot transact over — the honest design names which duplicate it tolerates.

  • Fanout replay after crash. Resolved by the checkpointed cursor plus the dedup gate: re-emitted messages from the last uncommitted page are dropped downstream, so resuming is safe and costs at most one page of rework.

  • Preference change mid-fanout. A user disables SMS while the 8-minute SMS drain is in progress. The fanout already read their preference and emitted (or not) the message; a late opt-out does not un-send a queued message. Acceptable — preferences are read-time snapshots, and the window is minutes. A hard unsubscribe (compliance-critical) is instead enforced at send time by a final check for the sms channel, trading a second Redis read for legal correctness on the one channel that demands it.

  • Rate-limit spiral under the burst. Without the token bucket, 1M SMS messages would fire faster than 2,000/s, the gateway returns 429s, retries pile onto the backlog, and each retry consumes capacity that should serve first-time sends — throughput collapses below 2,000/s under retry pressure. The token bucket prevents the spiral by never letting the send rate exceed the provisioned ceiling in the first place; 429s become rare and retries stay a trickle. Rate limiting here is overload prevention, not just fairness.

  • Frequency-cap race across notifications. Two notifications target the same user in the same second and both check the per-user cap before either increments it, so both pass and the user gets one over the cap. Resolved by making the cap check-and-increment atomic (a single Redis INCR against the sliding window with the limit compared on the returned value), so concurrent sends serialize on the counter.

  • Duplicate provider webhooks. Providers redeliver delivery receipts. Feedback Ingest dedups on the provider's message id before mutating status, so a redelivered "delivered" does not double-count the rolling counters.