Design a Distributed Notification System
Design a Distributed Notification System
Modern consumer and enterprise products deliver billions of notifications daily across Apple Push Notification service (APNs), Firebase Cloud Messaging (FCM), SMS aggregators (Twilio), and Transactional Email (SendGrid / AWS SES).
1. System Requirements
Functional Requirements
- Multi-Channel Dispatch: Support iOS/Android Push Notifications, SMS, Email, and in-app bell notifications.
- Templating Engine: Dynamically render notifications with user-specific localization, names, and promotional copy.
- User Notification Preferences: Respect granular user opt-in/opt-out preferences (e.g., disable marketing emails while keeping security SMS enabled).
- Rate Limiting & Anti-Spam: Throttle non-critical alerts to prevent overwhelming users (e.g. max 3 marketing pushes per user per day).
- Priority Scheduling: Urgent notifications (Two-Factor OTP codes, fraud alerts) bypass marketing queues and deliver in $< 3\text{ seconds}$.
Non-Functional Requirements
- High Throughput: Capable of processing 1 Billion notifications/day with peak surges of 100,000 notifications/sec.
- Resilience: Zero lost mission-critical alerts; automatic failover between third-party SMS/Email vendors.
- Observability: Real-time delivery status tracking (sent, delivered, opened, bounced).
2. High-Level Architecture Blueprint
+-----------------------+
| Microservices Engine |
| (Order, Auth, Fraud) |
+-----------+-----------+
|
| POST /api/v1/notifications
v
+-----------------------+
| Notification Gateway |
| (Auth, Validation) |
+-----------+-----------+
|
v
+-----------------------+
| Deduplication Filter |
| & User Opt-Out Check |
+-----+-----------+-----+
| |
+--------------------------+ +--------------------------+
| High Priority (OTP) | Normal / Bulk Marketing
v v
+------------------------+ +------------------------+
| Kafka Priority Queue | | Kafka Standard Queue |
+-----------+------------+ +-----------+------------+
| |
+-------------------+------------------+
|
v
+-----------------------+
| Worker Dispatch Pools |
+-----+-----+-----+-----+
| | | |
+-------------+ | | +-------------+
| | | |
v v v v
+------------+ +------------+ +------------+ +------------+
| Apple APNs | | Google FCM | | Twilio SMS | | AWS SES |
+------------+ +------------+ +------------+ +------------+3. Deep Dives & Edge Case Handling
Deep Dive 1: Deduplication via In-Memory Hashes
To prevent accidental loops where a buggy payment service sends 50 duplicate SMS receipts:
- Hash:
SHA256(user_id + channel + event_type + timestamp_minute). - The gateway attempts to store this key in Redis with
SET key 1 NX EX 300. If key already exists, the duplicate notification is silently discarded.
Deep Dive 2: Provider Failover & Circuit Breakers
If Twilio suffers an outage in Europe, SMS queues will back up.
- Workers monitor vendor error rates via a Resilience4j / Envoy Circuit Breaker.
- If error rate exceeds 10%, the system trips the breaker and routes European SMS traffic to a secondary vendor (e.g. Sinch or Infobip) within 500ms.
Key Architecture Trade-Offs & Decision Matrix
| Dimension | Option A: Asynchronous / Decoupled | Option B: Synchronous RPC | Trade-off & Recommendation |
|---|---|---|---|
| Communication Pattern | Message Queues (Kafka / SQS) | Direct gRPC / REST Calls | Message queues prevent cascading failures and provide backpressure buffering. |
| Failure Handling | Dead Letter Queue (DLQ) with Exponential Backoff | Immediate Circuit Breaker trip | DLQ preserves unparseable messages for manual operational inspection. |
| Ordering Guarantees | Partition-level Strict FIFO | Unordered Parallel Execution | Partition-key routing guarantees chronological order per entity without global lock contention. |
Related Topics
Expand your knowledge by learning about adjacent concepts in system design.
Event-Driven Architecture & Scaling with Apache Kafka
Design resilient, event-driven distributed architectures using Apache Kafka: topic partitioning, consumer group rebalancing, idempotence, and exactly-once semantics.
Kafka
Learn about how you can use Kafka to solve a large number of problems in System Design.
API Gateway
Master API Gateway architecture for distributed systems & FAANG interviews: Reverse Proxy vs Gateway vs Service Mesh, dynamic routing, edge security, JWT validation, distributed rate limiting, circuit breakers, and BFF patterns.
Cheat Sheet Utility
View and print a concise system design reference card.