Architectural Breakdown & AnalysisFAANG Aligned
A Hot Partition occurs when data or requests distribute unevenly, overloading a single partition node. For example, a celebrity user with 80 million followers posting a update.
1. Hybrid Fan-Out Architecture
- Fan-Out on Write (Push): Prepend posts to every follower's inbox cache. Excellent for standard users (fast read latency), but fails for celebrities (writing 80M entries crashes workers).
- Fan-Out on Read (Pull): Timelines are computed at query time by pulling posts from followed accounts. Saves write operations, but makes reads slow.
- Hybrid Pattern: Use Push for standard users (<10k followers). For celebrities, do not fan out on write. Instead, when followers load feeds, pull celebrity posts on-the-fly and merge them into the cached inbox.
2. Shard Key Salting
Append a random suffix to high-cardinality keys (CELEB_POST_1, CELEB_POST_2) to distribute write load across 10 distinct physical partitions, aggregating them during reads.
TIP — Real-World Example
Twitter/X uses a hybrid fan-out model to maintain low sub-100ms timeline rendering times while protecting backend DBs from write spikes.
Key Architectural Comparison
| Fanout Architecture | Ingestion Cost (Post Write) | Retrieval Cost (Feed Read) | Storage Footprint | Production Application |
|---|---|---|---|---|
| Fanout-on-Write (Push) | $O(N)$ where $N$ is follower count (expensive for celebrities). | $O(1)$ fast Redis read ($<20\text{ms}$). | High (stores post ID in millions of feeds). | Regular users (<25k followers). |
| Fanout-on-Read (Pull) | $O(1)$ single write to author timeline. | $O(K)$ merge-sort of followed creators at read time. | Minimal (stores post once). | High-follower accounts (>100k followers). |
| Hybrid Fanout | Push for standard users, Pull for celebrities. | Fast $O(1)$ feed load merged with celebrity posts. | Balanced. | Twitter, Instagram, LinkedIn. |