00. Design a Multi-Channel Notification System¶

~20 min read · Level: intermediate · Part 1 of 4 (Overview → HLD → LLD → Q&A)

Problem¶

A notification system takes a single request — "tell these users this thing" — and turns it into individual messages delivered over whatever channels each user has opted into: a push notification to their phone, an email, an SMS, sometimes all three. This is the machinery behind Slack's activity alerts, Uber's "your driver is arriving," a bank's fraud text, and a news app's breaking-news banner. One product event fans out into millions of individual deliveries, each shaped by that recipient's preferences, language, and device, and each handed to an external provider — Apple's APNs, a mail service, a telecom SMS gateway — that the system does not control and cannot fully trust.

What makes this a real system-design problem is not the sending of any one message; a single HTTP call to a provider is trivial. It is the fanout — one input becoming millions of outputs — combined with three facts that fight each other: the external providers rate-limit you and fail independently, users must never be double-notified, and some notifications are urgent while others can wait. Get the fanout, the dedup, and the rate control right and the rest is plumbing.

To keep the reasoning concrete, thread one scenario through the whole design: a breaking-news alert that must reach 10,000,000 subscribers within minutes, across push, email, and SMS, respecting each user's channel preferences, and never double-sending. After preference filtering that single alert becomes roughly 14 million individual messages — 9M push, 4M email, 1M SMS — and a duplicate on the SMS channel alone would waste thousands of dollars and erode the trust that makes people leave alerts on. That one alert, and the tension between "deliver in minutes" and "deliver each message exactly once at bearable cost," tests every decision below.

Functional requirements¶

Send to a user: given a user and a notification payload, deliver it over the channels that user has enabled.
Fan out to an audience: given a segment or list (up to tens of millions), expand it into per-user, per-channel messages.
Multi-channel with preferences: honor per-user, per-category channel opt-ins, quiet hours, and frequency caps.
Templating: render a notification from a versioned, channel-specific template with per-locale variants and payload substitution.
Delivery tracking: record per-message status (queued, sent, delivered, failed, bounced) and expose it for querying.
Retry and dedup: retry transient provider failures, and guarantee a user is never sent the same notification twice.

De-scoped for this round, and worth saying out loud so the interviewer hears it as a choice rather than an oversight: in-app inbox rendering and read-receipt UI, rich two-way messaging (replies, chatbots), the marketing-campaign scheduling and A/B-testing surface, and building the SMS/push transport itself — we integrate existing providers rather than terminate carrier or APNs protocols. These sit beside the core and do not change its architecture.

Non-functional requirements¶

The dominant constraint is fanout throughput under independent, rate-limited external providers. Everything downstream follows from that — the queue topology, the dedup strategy, the failover design.

Throughput: absorb a single request that expands to tens of millions of messages and drain it within the SLA of the notification's priority. The 14M-message alert must clear in minutes, not hours.
No duplicates: a given (notification, user, channel) is delivered at most once as far as the user perceives, even across retries and provider failover. This is a correctness requirement, not a nicety — duplicate fraud texts and double breaking-news buzzes are the failures users remember.
Prioritization: an urgent notification (OTP, fraud alert, breaking news) must not sit behind a bulk marketing blast. Latency requirements differ by two orders of magnitude across categories.
Availability of ingestion: accepting a send request must stay up even when a downstream provider is degraded; the request is durably queued and delivered on the provider's recovery, not rejected.
Cost-awareness: SMS costs real money per message (~$0.0075 each in the US); push and email are near-free. The design must let cost, not just latency, shape channel and retry decisions.
Eventual delivery, eventual status: it is acceptable for a low-priority notification to land seconds or minutes late, and for delivery status to lag further as provider webhooks trickle in.

Scale estimation¶

Take the breaking-news alert as the sizing event. The audience is 10,000,000 subscribers, but not every subscriber wants every channel. Assume opt-in rates for this high-priority category of 90% push, 40% email, 10% SMS. That expands to 9M + 4M + 1M = 14M individual messages from one API call — the fanout amplification factor is 1.4× the audience because many users get more than one channel.

The delivery target is "within minutes." Size for 5 minutes = 300 seconds. Aggregate throughput is 14M / 300 ≈ 47,000 sends/second, split by channel into 9M/300 ≈ 30,000/s push, 4M/300 ≈ 13,300/s email, and 1M/300 ≈ 3,300/s SMS. Push and email providers absorb their shares comfortably (APNs and a production email service handle tens of thousands per second over multiplexed connections). SMS is the wall: a telecom gateway typically caps a sender at a few hundred to a couple thousand messages per second even with a provisioned high-throughput short code. At a provisioned 2,000 SMS/s, 1M / 2,000 = 500 s ≈ 8.3 minutes — the SMS tail overruns the 5-minute target by design, and that is the honest bottleneck to name up front rather than pretend away.

Put this against steady state so the burst is in proportion. A mid-size platform might average 2 billion notifications/month, which is 2e9 / (30 × 86,400) ≈ 770/second average, or roughly 7,700/s at a 10× daily peak. The breaking-news burst at 47,000/s is about 6× the normal peak, concentrated into a few minutes — so the system is sized not for the average but for the spike, and the spike is a fanout event, not a gradual ramp.

Storage falls out of the message count. Per-message delivery status is ~200 bytes (ids, channel, state, timestamps, provider response code); at 2B/month that is 2e9 × 200 B ≈ 400 GB/month of append-only event data — a columnar or time-series store, not the operational database. The dedup keys are the interesting one: for the alert, 14M keys of ~100 bytes each held for a 72-hour retry window is 14M × 100 B ≈ 1.4 GB in an in-memory store per large alert — cheap, and the price of never double-sending.

Cost reconciles the channel mix. That one alert costs roughly 1M × $0.0075 = $7,500 in SMS, 4M × $0.0001 = $400 in email, and ~$0 in push — **~$7,900 total, of which 95% is SMS**. A dedup failure that resends the SMS channel does not just annoy a million people; it burns another $7,500. Cost is why dedup is a first-class requirement and not an afterthought.

API sketch¶

POST /v1/notifications
  body: {
    "template_id": "breaking_news_v3",
    "audience": { "segment": "alerts_optin" },   // or "user_ids": [...]
    "channels": ["push", "email", "sms"],          // allowed set; per-user prefs filter within
    "priority": "high",                             // high | default | bulk
    "data": { "headline": "...", "url": "..." },
    "idempotency_key": "alert-2026-07-03-quake"     // collapses duplicate submits
  }
  202: { "notification_id": "ntf_9f2c", "estimated_recipients": 10000000 }

GET /v1/notifications/{id}
  200: { "status": "delivering",
         "counts": { "queued": 14000000, "sent": 9.2e6, "delivered": 8.8e6, "failed": 41000 } }

PUT /v1/users/{user_id}/preferences
  body: { "channels": { "push": true, "email": true, "sms": false },
          "quiet_hours": { "tz": "Asia/Kolkata", "from": "22:00", "to": "07:00" },
          "frequency_cap": { "marketing": "3/day" } }

POST /v1/templates
  body: { "name": "breaking_news",
          "variants": { "push": {...}, "email": {...}, "sms": {...} },
          "locales": ["en", "hi", "es"] }

GET /v1/users/{user_id}/notifications      # inbox / history for in-app rendering

Solutioning¶

Start from the fanout and the shape of the system is forced. One API call must become 14 million deliveries, and those deliveries hit external providers at wildly different rates and reliabilities. The first move is to accept the request, durably record it, and return immediately — a 202, not a blocking send. A synchronous design that fanned out and sent inline would tie the caller's request latency to the slowest provider and could never sustain 47,000/s. So ingestion is fast and durable; the actual expansion and sending happen asynchronously behind a queue. The reframing that carries the whole design: 10 million subscribers is not a delivery-speed problem; it is a fanout-amplification problem — one request becoming 14 million independent, individually-preference-filtered, individually-retryable sends. You do not scale the send; you scale the fanout and the queue behind it.

The defining tradeoff is exactly-once delivery versus cost and complexity, and the honest answer is that exactly-once is a fiction here. The moment a message leaves for APNs or a telecom gateway, the system cannot know whether a timeout meant "not delivered" or "delivered but the ack was lost" — the provider is outside any transaction it can run. So delivery is at-least-once, made safe with dedup: every message carries a deterministic idempotency key (notification_id, user_id, channel), and a dedup store records which keys have been sent. A retry re-checks the key and skips a send that already happened. This costs the ~1.4 GB dedup store per large alert and a lookup per send, but it prevents the $7,500 double-SMS and the double-buzz. An attempt at true exactly-once — distributed transactions spanning an external provider — would add latency, still not cover provider-side duplicates, and buy nothing the dedup key does not. Not exactly-once delivery, but at-least-once plus dedup.

The second tradeoff is synchronous versus asynchronous delivery, resolved by priority rather than globally. Bulk and breaking-news traffic goes fully async through per-channel queues, because throughput matters more than the last hundred milliseconds. But a login OTP is different: the user is staring at a screen waiting for it, and a five-second queue delay is a failed login. So a narrow class of transactional, latency-critical notifications takes a priority express path — its own high-priority queue partition (or a near-synchronous send with a tight timeout) so it never queues behind a 14-million-message blast. The system is not "sync" or "async"; it is async with a fast lane, sized so the breaking-news burst cannot starve the OTP that arrives during it.

The third tradeoff is per-channel provider failover versus duplicate risk. Each channel has a primary and a secondary provider (APNs with a fallback path, two email vendors, two SMS gateways) because any single provider will have an outage measured in minutes to hours, and a notification system that goes dark when one vendor does has failed its one job. Failover introduces its own duplicate hazard — the message may have actually gone out via the primary just before you gave up and retried via the secondary — which is precisely why the dedup key is keyed on (notification, user, channel) and not on the provider: switching providers must not switch identities, or dedup would miss the duplicate. The cost is that the secondary provider is often pricier and the health-check-and-switch logic is real complexity, paid for by staying up through the vendor outages that are certain to come.

The result is a system whose front door is a fast, durable ingestion that fans an audience into per-user, per-channel, preference-filtered, template-rendered messages; whose middle is a set of priority-aware, per-channel queues; and whose edge is a fleet of channel workers that rate-limit against each provider, fail over between providers, and consult a dedup store so at-least-once never becomes visibly more-than-once. The following files take each decision down to components (HLD) and then to schemas, the fanout algorithm, and the concurrency corners (LLD).