01. Multi-Channel Notification System — High-Level Design¶

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the solutioning narrative into concrete boxes and the flows between them. Read the architecture top to bottom, follow a send request through fanout to a provider, then look at what happens when a provider or the dedup store fails under the breaking-news burst.

Architecture¶

   caller (service / campaign tool)
            │  POST /v1/notifications  (202, durable)
            ▼
   ┌────────────────────┐        ┌──────────────────┐
   │  Ingestion API      │───────▶│  Notification    │  system of record:
   │  (validate, dedup   │        │  metadata store  │  campaigns, status
   │   the submit, 202)  │        └──────────────────┘
   └─────────┬───────────┘
             │ enqueue "fanout job"
             ▼
   ┌────────────────────┐   reads   ┌──────────────────┐
   │  Fanout Service     │◀─────────▶│ Preference Svc   │ (per-user channel
   │  (audience → users, │           │  + Template Svc  │  opt-ins, quiet
   │   filter, render,   │◀─────────▶│                  │  hours, caps;
   │   emit per-channel) │           └──────────────────┘  rendered payloads)
   └─────────┬───────────┘
             │ one message per (user, channel), with idempotency key
             ▼
   ┌───────────────────────────────────────────────────────┐
   │        Message Bus  (Kafka, partitioned by channel      │
   │        × priority; e.g. push.high, email.bulk, sms.high)│
   └───┬───────────────┬───────────────────┬─────────────────┘
       ▼               ▼                   ▼
 ┌───────────┐   ┌───────────┐       ┌───────────┐
 │  Push     │   │  Email    │       │  SMS      │   channel workers:
 │  worker   │   │  worker   │       │  worker   │   rate-limit, dedup-check,
 └────┬──────┘   └────┬──────┘       └────┬──────┘   send, retry, DLQ
      │               │                   │
   ┌──┴───────────────┴───────────────────┴──┐    ┌──────────────┐
   │  Rate Limiter (per provider / per user)  │    │ Dedup Store  │
   └──┬───────────────┬───────────────────┬──┘    │ (Redis, TTL) │
      ▼               ▼                   ▼         └──────────────┘
 ┌─────────┐    ┌─────────┐         ┌─────────┐
 │APNs/FCM │    │SES/…    │         │Twilio/… │  external providers
 │(+backup)│    │(+backup)│         │(+backup)│  (primary + secondary)
 └────┬────┘    └────┬────┘         └────┬────┘
      │  delivery receipts / bounces (webhooks)
      ▼               ▼                   ▼
   ┌──────────────────────────────────────────┐
   │  Feedback Ingest ─▶ Status/Analytics store │ (append-only, columnar)
   └──────────────────────────────────────────┘

Read it top to bottom. A caller posts a send; the Ingestion API validates it, dedups the submission itself against an idempotency key, records it in the notification metadata store, and returns 202 — the request is now durable and the caller is done. Ingestion enqueues a fanout job. The Fanout Service expands the audience into individual users, consults the Preference Service to drop channels the user disabled and apply quiet hours and caps, asks the Template Service to render the payload per channel and locale, and emits one message per surviving (user, channel) onto the Message Bus, partitioned by channel and priority. Channel workers consume their partitions, check the Dedup Store, pass through the Rate Limiter, and hand the message to an external provider (failing over to a secondary if the primary is unhealthy). Providers later call back with delivery receipts and bounces, which Feedback Ingest folds into the status store. The critical insight the diagram encodes: sending is horizontal fanout, and every stage after ingestion is asynchronous and independently scalable.

Components¶

Ingestion API. The fast, durable front door. It validates the request, resolves the audience reference enough to estimate recipients, dedups the submission (an idempotency key on the API call so a client retry of POST /v1/notifications does not trigger two fanouts), writes a campaign/notification row, enqueues a fanout job, and returns 202. It never sends; it never blocks on a provider. This is what keeps ingestion available when providers are down.

Fanout Service. The workhorse. It turns an audience reference into a stream of individual users (paging through a segment that may hold tens of millions), and for each user applies preferences, renders templates, and emits per-channel messages. It is checkpointed and resumable so a crash mid-expansion does not restart 14 million messages from zero, and it runs many workers in parallel, each owning a slice of the audience.

Preference Service. The authority on what a user will accept: per-category channel opt-ins, quiet hours in the user's timezone, and frequency caps. It is read once per user during fanout, so it must serve reads at fanout throughput; it is backed by a low-latency KV store and heavily cached. Getting this wrong sends a notification a user explicitly muted — a compliance and trust failure, not a bug.

Template Service. Versioned, channel-specific templates with locale variants. breaking_news_v3 has a push variant (title + body, length-limited), an email variant (subject + HTML), and an sms variant (160-char plain text), each with en/hi/es translations. Rendering substitutes the request's data into the chosen variant. Versioning matters: a template edit mid-campaign must not change messages already rendered and queued.

Message Bus. A partitioned, durable log (Kafka or equivalent). Topics are split by channel × priority — sms.high, email.bulk, and so on — so a bulk email blast physically cannot occupy the SMS-high partition an OTP needs, and so each channel's consumers scale to that channel's provider throughput independently. Durability here is what lets ingestion return before delivery: the message is safe on the log.

Channel workers. Per-channel consumer fleets. Each worker pulls a message, checks the dedup store, acquires a rate-limit token for the target provider, sends, interprets the provider response (success, retryable error, permanent bounce), and either marks the dedup key sent, schedules a retry, or routes to a dead-letter queue. Workers are where provider-specific quirks live, behind a uniform adapter interface.

Rate Limiter. Enforces two independent limits: the provider's ceiling (do not exceed the SMS gateway's 2,000/s or it starts rejecting with 429s), and the per-user frequency cap (do not send one user twenty things in a minute even across notifications). It is a distributed token-bucket keyed by provider and by user, backed by the same in-memory store as dedup.

Dedup Store. An in-memory store (Redis) mapping the idempotency key (notification_id, user_id, channel) to a sent marker with a TTL covering the retry window (~72h). This is the single mechanism that keeps at-least-once from becoming visibly more-than-once.

Feedback Ingest + Status store. Providers report delivery asynchronously via webhooks — "delivered," "bounced," "user unsubscribed." Feedback Ingest consumes these, updates per-message status, and feeds an append-only columnar store for analytics and the status API. Bounces and unsubscribes also write back to the Preference Service, so a hard-bounced email stops being retried.

Primary write path (a send request → deliveries)¶

POST /v1/notifications reaches the Ingestion API. It checks the submission idempotency_key; if this exact request already exists, it returns the existing notification_id (no second fanout).
It writes a notification row (status = fanning_out, audience reference, channel set, priority, rendered-template pointer) to the metadata store and enqueues a fanout job, then returns 202 with an estimated recipient count.
The Fanout Service picks up the job and pages through the audience. For each user it calls the Preference Service: drop channels the user disabled for this category, skip users in quiet hours (unless the notification is override-urgent), and check frequency caps.
For each surviving (user, channel), it renders the channel/locale template and emits a message — carrying the deterministic idempotency key — onto the bus partition for that channel and priority.
The relevant channel worker consumes the message, checks the Dedup Store (SET key NX — if already present, drop as a duplicate), acquires a rate-limit token for the provider, and sends.
On provider 2xx, it marks the dedup key sent and records status = sent. On a retryable error, it schedules a backed-off retry (the dedup key is not marked sent, so the retry proceeds). On a permanent error, it routes to the DLQ and records failed.
Later, the provider's webhook arrives at Feedback Ingest and promotes sent → delivered (or bounced), updating the status store.

Primary read path (status and inbox)¶

GET /v1/notifications/{id} hits the status API, which reads aggregate counts (queued/sent/delivered/failed) for the notification. These are maintained as rolling counters updated by workers and feedback ingest, not computed by scanning 14M rows on each call.
GET /v1/users/{id}/notifications serves the in-app inbox from the per-user notification history — a straightforward keyed read, separate from the delivery pipeline so inbox queries never touch the hot send path.
Delivery receipts flow the other way on the read side: provider webhooks are themselves reads-from-outside, deduplicated (providers redeliver webhooks) before they mutate status.

Storage choices¶

Notification / campaign metadata: relational or document store. Modest volume (one row per send request), needs transactional updates to status and lifecycle, and is queried by id. A relational database or document store fits; it is the system of record for "what was requested."
User preferences: low-latency KV store. Keyed by user_id, read once per user per fanout at up to tens of thousands per second, so it must be fast and cacheable. A KV store (DynamoDB/Cassandra) or a heavily-cached relational table works; the access pattern is a point read, occasionally a write from the settings screen or a bounce.
Templates: versioned store + object storage. Small, read-heavy, mutated rarely. Metadata in a relational table, large HTML bodies in object storage, every version immutable so in-flight campaigns are stable.
Message bus: partitioned durable log (Kafka). Chosen for durability (ingestion returns before delivery), ordered partitions, consumer-group scaling, and replay after a consumer outage. Partitioned by channel × priority as described.
Dedup + rate-limit + counters: in-memory store (Redis). Microsecond SET NX for dedup, token buckets for rate limiting, atomic increments for status counters. Volatile by design — a dedup key is only needed for the retry window.
Delivery status / analytics: append-optimized columnar store. 400 GB/month of append-only events queried by aggregation over time, channel, and provider — a columnar/time-series store (ClickHouse, BigQuery), kept off the operational path so analytics load never touches delivery.

Scaling¶

Fanout path. Fanout is embarrassingly parallel: partition the audience and run N fanout workers, each expanding its slice. For the 10M-subscriber alert, splitting the audience into 100 slices of 100k users lets 100 workers each emit ~140k messages; at a few thousand emits/second per worker the whole expansion completes in well under a minute, and the message bus — not the fanout — becomes the pace-setter. Scaling the fanout is adding workers; the only shared dependency is the Preference Service, which is why it is cached hard.

Delivery path. Each channel scales to its provider's ceiling independently. Push at 30,000/s is fine — APNs/FCM multiplex over HTTP/2 and the worker fleet is CPU-bound on serialization, so add workers until the provider connection is saturated. Email at 13,300/s sits under a provisioned 5,000/s-per-region service by spreading across regions or accepting a slightly longer tail. SMS is the hard ceiling: at a provisioned 2,000 SMS/s, 1M messages take ~500 s regardless of how many workers you add, because the limit is the gateway, not your compute. Adding SMS workers past that point just makes them wait on rate-limit tokens. Scaling SMS throughput means provisioning more numbers/short codes or negotiating carrier throughput — a procurement problem, not a code problem.

Hot dependency: the dedup store. At the 47,000/s aggregate peak, that is 47,000 SET NX operations per second against Redis plus rate-limit checks — well within a small cluster, but it is the one component every send touches, so it is clustered and sharded by key to spread load and survive a node loss.

Operational signals¶

The healthy signal is per-channel queue lag trending to zero at the target drain rate — messages enter, workers pull them at the provider's sustainable rate, and lag stays flat or shrinks. The first metric to degrade under trouble is consumer lag on the slowest channel's high-priority partition, almost always SMS: when a provider throttles or slows, its workers block on rate-limit tokens and that partition backs up first while push and email stay clear. The misleading metric is the "sent" count — it climbs steadily and looks healthy even while a provider is silently soft-bouncing or dropping messages, because "we handed it to the provider" is not "the user received it"; the number that tells the truth is the delivered rate from webhooks, which lags and can diverge sharply from sent. The graph an experienced operator opens first during an incident is per-provider error/429 rate alongside that provider's queue lag — a rising 429 rate with growing lag says the provider is throttling you and the fix is backpressure or failover, whereas rising lag with a flat error rate says your workers are under-provisioned and the fix is more consumers (up to the provider ceiling).

Failure modes and resilience¶

Provider outage (e.g. APNs down mid-alert). The channel's workers see connection failures. Mitigation: health-check-driven failover to the secondary provider, with the dedup key keyed on (notification, user, channel) — not on provider — so a message that may have squeaked out via the primary before failover is not re-sent via the secondary. Messages that fail both providers hold in the queue and retry on recovery; ingestion stays up throughout.
Provider rate-limiting under the breaking-news burst. This is the scenario's signature failure. The alert dumps 1M SMS onto a gateway provisioned for 2,000/s. Without control, workers fire faster than the ceiling, the gateway returns 429s, and naive retries amplify the overload into a spiral. Mitigation: the rate limiter's token bucket paces sends to exactly the provisioned rate, so the SMS partition drains in a controlled ~8 minutes rather than melting down; the queue absorbs the backlog durably; and high-priority SMS (fraud OTPs arriving during the alert) ride a separate sms.high partition so they are not stuck behind the million-message blast.
Fanout crash mid-expansion. A worker dies after emitting 3M of its slice's messages. Mitigation: fanout is checkpointed by audience cursor, so on restart it resumes from the last committed cursor; messages already emitted are harmless duplicates because the dedup key drops them at the worker.
Dedup store outage. The dangerous one. If Redis is down, workers cannot check for duplicates. Mitigation: the policy is a deliberate choice per channel — for cheap, low-harm channels (push) fail open (send, accept rare duplicates) to preserve delivery; for expensive or high-trust channels (SMS, email) fail closed (pause the channel) rather than risk a $7,500 double-send. The store is clustered with replicas so full loss is rare.
Duplicate provider webhooks. Providers redeliver delivery receipts. Mitigation: Feedback Ingest dedups webhooks by provider message id before mutating status, so a redelivered receipt does not double-count.
Poison messages. A malformed message that crashes a worker on every attempt. Mitigation: a retry ceiling routes it to the DLQ after N attempts, so one bad message cannot stall a partition, and the DLQ is inspected out of band.

Where this shows up in production¶

Twilio — vends SMS through a messaging service that pools many numbers and enforces per-number and account throughput, exactly the provider-side rate ceiling that makes SMS the fanout bottleneck here.
Apple APNs / Google FCM — accept push over multiplexed HTTP/2 connections and return per-token feedback (unregistered devices), the source of the bounce-to-preference writeback loop.
Amazon SES / SendGrid — production email with per-region send-rate quotas and asynchronous bounce/complaint notifications, the model for the email channel's rate limit and feedback ingest.
Uber — runs a priority-tiered notification platform so a "driver arriving" push is never queued behind a promotional blast, the channel × priority partitioning made concrete.
Slack — fans a single channel message into per-user notifications filtered by each user's notification preferences and Do-Not-Disturb windows, the Preference Service role at scale.
Airbnb / Netflix — internal notification platforms that centralize templating, preferences, and multi-channel delivery so product teams send one request instead of integrating providers themselves, the reason this is a platform, not a feature.
Amazon SNS — a managed pub/sub fanout that turns one publish into many endpoint deliveries, the message-bus fanout primitive underneath this design.
Kafka — the partitioned durable log used across the industry as the notification pipeline's backbone, giving replay-after-outage and per-partition consumer scaling.
Redis — the near-universal choice for the dedup key set and distributed token-bucket rate limiting, both on the hot send path.