03. Multi-Channel Notification System — Interview Q&A

~15 min read · Part 4 of 4 (Overview → HLD → LLD → Q&A)

These are the questions an interviewer actually asks once the diagram is on the board. Each answer is the strong version, followed by the wrong answer that quietly sinks candidates.

Q1. A breaking-news alert must reach 10 million subscribers in minutes. How does the system handle it? Recognize first that this is a fanout-amplification problem, not a delivery-speed problem: one API call becomes ~14 million individual messages once preferences fan it across 9M push, 4M email, and 1M SMS. Ingestion returns 202 immediately after durably recording the request and enqueuing a fanout job. The fanout service partitions the 10M audience into ~100 slices and expands them in parallel — preference-filtering, template-rendering, and emitting per-channel messages onto priority-partitioned queues — so the whole expansion lands on the bus in under a minute. Channel workers then drain each queue at its provider's sustainable rate: push and email clear in ~5 minutes, SMS in ~8 because the gateway caps at ~2,000/s. Every message carries a deterministic idempotency key so retries and queue redeliveries never double-send. Common wrong answer to avoid: "Loop over the 10M users and send inline." A synchronous loop ties the request to the slowest provider, can't sustain the throughput, and loses all progress on a crash.

Q2. Why not guarantee exactly-once delivery? Because the moment a message leaves for APNs or a telecom gateway, you're outside any transaction you can run — a timeout can't distinguish "not delivered" from "delivered but the ack was lost." Exactly-once across an external boundary you can't transact over is a fiction. The honest design is at-least-once made safe with dedup: a deterministic key (notification_id, user_id, channel) claimed atomically before each send, so retries and redeliveries drop as duplicates. That costs a ~1.4 GB dedup store for a 14M-message alert and one lookup per send, and it prevents the $7,500 double-SMS and the double-buzz. Common wrong answer to avoid: "Use a distributed transaction / two-phase commit across the provider." You can't enlist a third-party SMS gateway in your transaction, and even if you could it wouldn't catch provider-side duplicates.

Q3. Where does the dedup key come from, and why not include the provider in it? The key is deterministic from stable identity — notification_id : user_id : channel — so any worker, on any retry, computes the same key without coordination. It deliberately excludes the provider because failover must not mint a new identity: if the primary SMS gateway actually sent the message just before you gave up and failed over to the secondary, a provider-scoped key would treat the secondary send as new and the user gets two texts. Keying on channel, not provider, means the dedup gate still recognizes the duplicate across a failover. Common wrong answer to avoid: "Use a random UUID per attempt as the idempotency key." A per-attempt id makes every retry look unique, which defeats dedup entirely.

Q4. SMS is capped at ~2,000/second but you have 1M to send. What do you do? Accept that SMS has a hard provider ceiling and design around it rather than pretend it away: 1M / 2,000 = ~500 seconds, so the SMS tail is ~8 minutes even fully provisioned, while push and email finish in ~5. A token-bucket rate limiter paces sends to exactly 2,000/s so the gateway never sees a flood and starts rejecting; the durable queue holds the backlog; and high-priority SMS (a fraud OTP arriving during the alert) rides a separate sms.high partition so it isn't stuck behind 600k blast messages. Raising the ceiling is procurement — more numbers, short codes, or carrier throughput — not more compute; adding workers past the ceiling just makes them wait on tokens. Common wrong answer to avoid: "Add more SMS workers to go faster." Past the gateway ceiling, extra workers block on rate-limit tokens; the limit is the provider, not your fleet.

Q5. What happens if you send faster than the provider allows? It triggers a rate-limit spiral. The gateway returns 429s, naive retries pile back onto the queue, and those retries consume the same limited capacity that should serve first-time sends — so effective throughput collapses below the ceiling under retry pressure, and the backlog grows. The fix is prevention, not reaction: the token bucket never lets the send rate exceed the provisioned ceiling in the first place, so 429s stay rare and retries stay a trickle. Rate limiting here is overload prevention, not just fairness between tenants. Common wrong answer to avoid: "Retry aggressively on 429 until it succeeds." Aggressive retries are what turn a throttle into an outage; you need backoff plus a bucket that caps the source rate.

Q6. How do you keep an urgent OTP from getting stuck behind a bulk blast? Partition the message bus by channel and priority — sms.high versus sms.bulk, and so on — and give high-priority partitions dedicated consumer capacity. A login OTP or fraud alert lands on the high partition and drains immediately even while a 14M-message blast is still working through the bulk partition. For the most latency-critical transactional notifications you can go further with a near-synchronous express path and a tight timeout. The system isn't globally sync or async; it's async with a fast lane, sized so the breaking-news burst can't starve the OTP that arrives during it. Common wrong answer to avoid: "One FIFO queue for everything." FIFO means the OTP waits behind however many million messages were enqueued first — a failed login during every large campaign.

Q7. A user has SMS muted globally but should still get breaking-news texts. How do preferences resolve? Preferences are layered: a category-level override sits above the global channel flag. Resolution is category-first — check categories[breaking_news][sms] and honor it if set — then fall back to the global channel opt-in as the default. So a user who muted SMS globally but opted into breaking-news SMS gets the alert, while their marketing SMS stays off. Quiet hours and frequency caps apply on top, with an override_urgent flag that lets a genuine emergency pierce quiet hours. Get the ordering wrong and you either spam muted users or drop alerts they explicitly wanted. Common wrong answer to avoid: "One boolean per channel." A single global flag can't express "SMS off except for emergencies," which is exactly the case that matters for high-trust notifications.

Q8. The dedup store (Redis) goes down mid-alert. What happens? Workers can no longer check for duplicates, so you choose a policy per channel rather than one blanket behavior. For cheap, low-harm channels like push, fail open — keep sending and accept rare duplicates, because a missed breaking-news push is worse than a rare double-buzz. For expensive or high-trust channels like SMS and email, fail closed — pause the channel — because a dedup-blind blast risks a $7,500 double-SMS and a wave of duplicate texts users will remember. The store runs clustered with replicas so total loss is rare, but the design states its failure policy explicitly instead of assuming the store is always up. Common wrong answer to avoid: "Just keep sending; Redis will come back." Blindly sending without dedup on the SMS channel is how you bill twice and erode trust in one incident.

Q9. What's the difference between "sent" and "delivered," and why does it matter operationally? "Sent" means the worker handed the message to the provider and got a 2xx; "delivered" means the provider's asynchronous webhook later confirmed it reached the device or inbox. They diverge — a provider can accept a message and then soft-bounce it, drop it, or have it filtered as spam. The trap is watching the sent count during an incident: it climbs steadily and looks healthy while delivery is quietly failing. The metric that tells the truth is the delivered rate from webhooks, which lags but reflects reality. Operationally, an expert watches per-provider error/429 rate alongside queue lag to distinguish "provider is throttling us" (backpressure or fail over) from "we're under-provisioned" (add workers). Common wrong answer to avoid: "Track sent count for success." Sent is an optimistic proxy; a provider outage can leave sent climbing while delivered flatlines.

Q10. How do you handle a provider outage without duplicating or dropping messages? Each channel has a primary and secondary provider with health-check-driven failover; when the primary fails, workers route to the secondary and messages that fail both hold in the durable queue for retry on recovery. Ingestion stays up the whole time because it never depends on a provider being healthy. The duplicate hazard is real — a message may have gone out via the primary just before failover — so the dedup key is scoped to (notification, user, channel) and not the provider, so a failover send of an already-sent message is recognized and dropped. Drops are handled by durability: nothing leaves the queue until a worker confirms a terminal state. Common wrong answer to avoid: "Fail over to the backup and resend everything in flight." Blind resend on failover double-sends everything the primary already delivered unless dedup is provider-agnostic.

Q11. How does the fanout survive a crash halfway through 10 million users? The fanout is partitioned into slices and each worker checkpoints its position — the audience cursor — after every page of ~1,000 users. On crash and restart, the worker resumes from the last committed cursor instead of re-expanding its whole slice from zero, so it re-emits at most one page's worth of messages. Those re-emitted messages are harmless: they carry the same deterministic idempotency keys and get dropped by the dedup gate at the worker. So the expensive expansion is resumable and the safety net for the overlap is the same dedup that protects retries. Common wrong answer to avoid: "Restart the whole fanout job." Re-expanding 10M users from scratch wastes minutes of work and, without dedup, would double-send everything already emitted.

Q12. Would you store per-message delivery status in your main database? No. That's ~14M rows for one alert and 400 GB/month of append-only events at steady state, queried by aggregation over time, channel, and provider — the wrong shape and volume for the operational database, and mixing it in lets analytics load degrade the send path. Emit delivery events to an append-optimized columnar store (ClickHouse, BigQuery) and keep only the notification-level rolling counts (queued/sent/delivered/failed) in the operational store, maintained as atomic counters rather than computed by scanning millions of rows per status query. Common wrong answer to avoid: "One row per message in the same relational table, updated on each status change." That's tens of millions of hot-updated rows per campaign contending with the ingestion path.

Deeper follow-ups

• How would you enforce a hard, compliance-grade unsubscribe (e.g. a user replies STOP to an SMS) so it takes effect immediately, even for messages already fanned out and queued?
• How would you batch or collapse notifications so a user who triggers 50 events in a minute gets one digest instead of 50 pushes?
• The same alert must go out in 12 languages — where does locale resolution happen, and how do you keep template rendering off the hot send path?
• How would you A/B-test two versions of a template across a segment without breaking the "pinned template version" guarantee for in-flight sends?
• If a downstream consumer (analytics) is down for an hour, how does the pipeline recover the status events without losing them?
• How would you extend the priority model to guarantee a latency SLA per tier (e.g. OTP p99 under 3 seconds) while a 14M blast is draining?

How this round is scored

Interviewers use the notification system to see whether you recognize fanout as the core problem and refuse to treat it as "just send a message." The strongest early signal is naming the amplification — one request becoming 14 million preference-filtered sends — and building async ingestion, parallel fanout, and priority-partitioned queues around it rather than a synchronous loop. Seniority shows up in the delivery-semantics discussion: candidates who assert "exactly-once" reveal they haven't shipped against real providers, while those who explain at-least-once-plus-dedup, and can name which duplicate they tolerate per channel in the timeout-ambiguity corner, have run this in production. The rate-control answer separates the same two groups — knowing that the SMS gateway, not your fleet, is the ceiling, and that a token bucket is overload prevention, not fairness. Doing the fanout and cost math out loud (14M messages, $7,900 with 95% in SMS, ~8-minute SMS tail) and using it to justify the dedup investment and the channel-by-channel failure policies is what pushes an answer from "correct" to "senior."