Design a News Aggregator (Google News)
Design a News Aggregator (Google News)
Google News ingests hundreds of thousands of articles published daily across thousands of media outlets, clusters stories covering the exact same event into a single headline card, and personalizes feeds to reader interests.
1. System Requirements
Functional Requirements
- Content Crawling: Ingest news articles via publisher RSS feeds, sitemaps, and direct web crawling.
- Deduplication & Clustering: Group disparate articles describing the same breaking news story into one cluster.
- Category Tagging: Classify articles into sections (World, Business, Technology, Science).
- Personalized Feed: Rank stories based on freshness, publisher authority, and reader historical clicks.
Non-Functional Requirements
- Freshness: Breaking news stories must appear in user feeds within $< 5\text{ minutes}$ of publication.
- High Read Throughput: Serve 50,000+ QPS of personalized feed views with low latency ($< 150\text{ms}$).
2. Article Deduplication via SimHash & MinHash
When multiple journalists cover a White House press release, their articles share 70%+ identical phrasing.
- SimHash: Generates a 64-bit fingerprint of an article where similar text generates similar hash bit patterns (unlike cryptographic hashes like SHA-256 where 1 bit change flips 50% of bits).
- Hamming Distance: If the Hamming distance between two SimHash fingerprints is $\le 3\text{ bits}$, the articles are marked as duplicates and merged into the same cluster.
Key Architecture Trade-Offs & Decision Matrix
| Decision Factor | Choice A | Choice B | Selected Architectural Rationale |
|---|---|---|---|
| Communication Protocol | REST over HTTP/2 | WebSockets / gRPC | Use REST for stateless CRUD APIs; use WebSockets for low-latency bidirectional streaming. |
| State Management | Centralized Distributed Store | Local Memory with Sticky Sessions | Centralized Redis / Database state allows seamless container auto-scaling and zero-downtime rolling deploys. |
| Concurrency Control | Optimistic Locking (version column) | Pessimistic Locking (SELECT FOR UPDATE) | Optimistic locking delivers superior throughput for low-conflict consumer operations. |
Related Topics
Expand your knowledge by learning about adjacent concepts in system design.
Design a URL Shortener (TinyURL / Bitly Architecture)
Design a scalable URL shortening service capable of handling billions of redirects: Base62 encoding, distributed ID generation, database indexing, and 301 vs 302 redirect trade-offs.
Design Bitly (Scalable URL Shortener & Analytics Platform)
Design a high-scale URL shortening and click tracking service like Bitly or TinyURL capable of handling billions of redirects with sub-15ms latency.
Dropbox
System design answer key for designing a file storage and synchronization service like Dropbox, built by FAANG managers and staff engineers.
Cheat Sheet Utility
View and print a concise system design reference card.