How Discord Stores Trillions of Messages: The Migration from MongoDB to Cassandra to ScyllaDB\n\nDiscord serves over 200 million monthly active users exchanging billions of messages daily across millions of private voice and text channels. Maintaining predictable sub-millisecond query latencies across trillions of messages is one of the most celebrated database engineering case studies in modern infrastructure.\n\n---\n\n## 1. Chapter 1: The MongoDB Era (2015)\n\nIn Discord's earliest days, message data was stored in a single MongoDB replica set.\n\n### Why MongoDB Failed\n* RAM Saturation: MongoDB required entire indexes to reside in memory for fast performance. As message volume crossed 100 million rows, active indexes exceeded physical RAM.\n* Unpredictable Degradation: When queries missed the RAM working set, page faults caused disk read storms, spiking API latencies.\n* Global Write Locking: Older MongoDB versions struggled with concurrency under continuous write load, forcing an immediate architectural overhaul.\n\n---\n\n## 2. Chapter 2: The Apache Cassandra Era (2017)\n\nDiscord migrated all chat history to an Apache Cassandra cluster. Cassandra's masterless distributed design and Log-Structured Merge (LSM) storage engine initially handled write throughput effortlessly.\n\n\n[Client App] ──► [Elixir Gateway] ──► [Rust Data Service] ──► [ScyllaDB Cluster]\n (Shard-per-Core)\n (Async Direct I/O)\n\n\n### The Cassandra Data Model\nMessages were partitioned by channel_id into a partitioned wide-column family:\nsql\nCREATE TABLE messages (\n channel_id bigint,\n bucket int,\n message_id bigint,\n author_id bigint,\n content text,\n PRIMARY KEY ((channel_id, bucket), message_id)\n) WITH CLUSTERING ORDER BY (message_id DESC);\n\n* Time-Bucketing: Because Discord snowflakes embed timestamps in their high bits, messages were grouped into 10-day buckets to keep partition sizes under Cassandra's 100 MB recommended ceiling.\n\n### Why Cassandra Hit the Wall\nAs message counts crossed trillions, serious operational failure modes emerged:\n1. JVM Garbage Collection (GC) Pauses: Stop-the-world JVM GC pauses routinely halted node operations for 1 to 3 seconds, triggering cascading read timeouts across the cluster.\n2. Tombstone Catastrophes: When users deleted messages or channels were purged, Cassandra wrote delete markers called tombstones. Scanning past thousands of tombstones during message retrieval caused read queries to timeout or trigger node OOM crashes.\n3. Hot Partition Storms: High-traffic announcement channels (e.g., in servers with 500,000 members) hammered single nodes, causing localized latency spikes that corrupted cluster-wide SLAs.\n\n---\n\n## 3. Chapter 3: The Migration to ScyllaDB (2022–Present)\n\nDiscord evaluated alternatives and migrated to ScyllaDB, a ground-up C++ rewrite of Cassandra designed around the Seastar asynchronous framework.\n\n### Core Advantages of ScyllaDB\n* Thread-per-Core (Shared-Nothing) Architecture: ScyllaDB assigns one OS thread per CPU core, pinned via hardware affinity. CPU cores communicate strictly via lock-free ring buffers, completely eliminating mutex contention.\n* Zero Garbage Collection: Written in C++, memory is allocated statically and managed explicitly. GC pauses dropped from 2,000ms to 0ms.\n* Direct I/O (Linux AIO / io_uring): ScyllaDB bypasses the OS page cache entirely, reading and writing straight to NVMe storage using asynchronous non-blocking system calls.\n\n---\n\n## 4. The Intermediate Rust Routing Tier (Request Coalescing)\n\nTo prevent hot channels from overwhelming database storage engines, Discord placed an intermediate Rust Data Service between the API layer and ScyllaDB:\n\n### The Singleflight / Coalescing Pattern\nIf 1,000 users in a massive gaming server simultaneously request the latest 50 messages of a channel:\n1. The Rust worker registers an in-flight query channel for (channel_id, bucket).\n2. The first request issues a read to ScyllaDB.\n3. The subsequent 999 concurrent requests subscribe to the same in-flight future.\n4. When ScyllaDB returns the result 4ms later, the single response is broadcast to all 1,000 callers simultaneously.\n\n> [!TIP] Production Result\n> Request coalescing reduced raw database read queries during viral channel events by over 95%.\n\n---\n\n## 5. Architectural Comparison\n\n| Architecture Vector | MongoDB (2015) | Apache Cassandra (2017) | ScyllaDB + Rust (Current) |\n| :--- | :--- | :--- | :--- |\n| Engine Language | C++ | Java (JVM) | C++ (Seastar Async) + Rust |\n| Concurrency | Thread pool + Locks | Multi-threaded + Mutexes | Thread-per-Core (Lock-Free) |\n| GC Latency Spikes | N/A (Memory thrash) | Frequent 1s–3s GC pauses | Zero GC Overhead |\n| p99 Read Latency | > 800ms (High load) | ~150ms – 2,000ms | 15ms Consistent |\n| Server Footprint | Monolithic Shards | 177 Cassandra nodes | 72 ScyllaDB nodes (2.5x capacity) |