How Discord Stores Trillions of Messages

Discord’s message storage story is mostly a lesson in how database bottlenecks change shape as workload and scale change. The path from MongoDB to Cassandra to ScyllaDB was not driven by fashion. Each move addressed a specific failure mode that became painful at the next order of magnitude.

The earliest version of Discord used a single MongoDB replica. That is a perfectly reasonable choice for a young product. Message data is document-shaped, development speed matters, and one replica keeps operations simple. The problem was not that MongoDB suddenly stopped working at one hundred million messages. The problem was working set pressure. Once the active data and its indexes no longer fit comfortably in memory, reads became more dependent on disk access. Tail latency gets ugly fast when a chat system serves many small, time-ordered reads and the database starts paging unpredictably.

A chat product has a very specific access pattern. Messages are appended constantly, but users mostly read recent history within a channel or conversation. That makes partitioning strategy critical. Cassandra was attractive because it is built for horizontal scale, high write throughput, and predictable ownership of partitions across nodes. Discord could model messages by channel and message identifier, spread load over a cluster, and avoid the single-node ceiling that was biting the MongoDB setup.

At the next stage, new failure modes appeared. Cassandra uses an LSM tree storage engine. That is good for writes because new data is batched and flushed sequentially, but it complicates reads. A read may need to consult multiple SSTables and reconcile tombstones, memtables, and compaction state before returning a result. For a messaging workload with hot channels and frequent fetches of recent history, that read amplification shows up as latency variance rather than a simple average slowdown.

Operational work makes the situation worse. Cassandra clusters need compaction, repair, streaming during rebalancing, and careful handling of large partitions. Those are not edge cases. They are part of normal life. Each maintenance task competes for I/O and CPU with production traffic. As the cluster grew to hundreds of nodes and trillions of messages, the cost of routine maintenance rose sharply. Even if median latency looked acceptable, tail behaviour during garbage collection pauses, compaction spikes, or hotspot traffic could make the user experience inconsistent.

That is where ScyllaDB enters the story. Scylla is wire-compatible with Cassandra, but its internals are designed around a shard-per-core model in C++ rather than a JVM process that relies on large garbage-collected heaps. Removing long GC pauses does not magically solve all data problems, but it does remove one common source of unpredictable tail latency. Better I/O scheduling and per-core ownership also help the system use hardware more efficiently under mixed workloads.

Discord’s broader redesign matters too. Public descriptions of the platform point to a monolithic API paired with a dedicated Rust data service in front of ScyllaDB. That kind of middle layer is useful because it constrains how storage is accessed. Instead of every product path constructing its own database queries, one service can standardise key layouts, pagination behaviour, backfill rules, and hot-path caching. In storage systems, a consistent access pattern is often as valuable as a faster database engine.

The deeper lesson is that write scalability is not the same as read predictability. Many systems can absorb large append rates. Far fewer remain easy to operate when the dataset is huge, hot partitions emerge, and users notice p99 latency rather than mean throughput. Discord’s database evolution shows why storage choices must be judged by compaction cost, repair burden, hotspot behaviour, and tail latency under maintenance, not just by whether they scale on paper.