Core Fundamentals• 35 Min
Scaling Writes (Sharding, Partitioning & Write-Ahead Queues)
HARD
Scaling Writes (Sharding, Partitioning & Write-Ahead Queues)
When write throughput exceeds the physical limits of a single database master's NVMe drive, systems must shard data across physical nodes and decouple writes through asynchronous queuing.
1. Database Sharding Strategies
- Range-Based Sharding: Partition data by alphabetical or numerical range (e.g. Users A–F on Shard 1, G–M on Shard 2).
- *Pitfall*: Causes unbalanced write hot spots on specific ranges.
- Hash-Based Sharding: Compute
hash(partition_key) % N_shardsto distribute rows evenly. - Directory-Based Sharding: A lookup service maps specific entities to shard IDs, enabling easy re-balancing.
2. Write Buffering & LSM-Trees
- Relational databases using B-Trees require random disk I/O, which degrades write speed.
- NoSQL databases (Cassandra, RocksDB) use Log-Structured Merge-Trees (LSM-Trees):
- Writes are appended sequentially to an in-memory MemTable and a commit log.
- Flushed sequentially to immutable SSTables on disk, turning slow random writes into fast sequential writes.
Key Architecture Trade-Offs & Decision Matrix
| Decision Factor | Choice A | Choice B | Selected Architectural Rationale |
|---|---|---|---|
| Communication Protocol | REST over HTTP/2 | WebSockets / gRPC | Use REST for stateless CRUD APIs; use WebSockets for low-latency bidirectional streaming. |
| State Management | Centralized Distributed Store | Local Memory with Sticky Sessions | Centralized Redis / Database state allows seamless container auto-scaling and zero-downtime rolling deploys. |
| Concurrency Control | Optimistic Locking (version column) | Pessimistic Locking (SELECT FOR UPDATE) | Optimistic locking delivers superior throughput for low-conflict consumer operations. |
Related Topics
Expand your knowledge by learning about adjacent concepts in system design.
Core FundamentalsMEDIUM
Consistent Hashing Architecture & Implementation
Deep dive into consistent hashing algorithms, virtual nodes, token rings, and minimal key remapping during cluster resizing in distributed databases.
Infrastructure & MessagingHARD
Event-Driven Architecture & Scaling with Apache Kafka
Design resilient, event-driven distributed architectures using Apache Kafka: topic partitioning, consumer group rebalancing, idempotence, and exactly-once semantics.
Core FundamentalsMEDIUM
Networking Essentials
Learn the important parts of networking that you'll need to know for your system design interviews
Cheat Sheet Utility
View and print a concise system design reference card.