4 Data Sharding Algorithms

Sharding splits one logical dataset across multiple physical partitions so storage, write throughput, and maintenance work can scale beyond a single machine. The tricky part is not dividing data once. It is choosing a placement rule that still works when the dataset grows, traffic skews, and machines need to be added or replaced.

Four sharding algorithms appear often because they optimise different operational constraints.

1. Range-based sharding

Range-based sharding assigns each shard a contiguous slice of key space. One shard may hold customer IDs from 1 to 1,000,000, another from 1,000,001 to 2,000,000, or a time-series system may split by day or month.

The main advantage is locality. Range scans stay efficient because nearby values live together. Archival tasks, time-window queries, and tenant exports often benefit from this. The placement rule is also easy to reason about during debugging.

The weakness is uneven traffic. If new writes always land in the latest timestamp range, the newest shard becomes a hotspot while older shards stay quiet. Once skew appears, rebalancing means moving large chunks of real data. Range sharding works best when access patterns are range-oriented and fairly predictable.

2. Hash-based sharding

Hash-based sharding applies a hash function to a shard key and uses the result to choose a shard. Sequential customer IDs, order numbers, or UUIDs become scattered across the fleet instead of piling up on one machine.

This is a strong default for point lookups and write-heavy workloads keyed by identity. It smooths out hotspots that would appear with naive range partitioning and usually needs less manual balancing.

The price is loss of locality. Queries that need range scans, sorted traversal, or grouping by nearby values often have to hit many shards. Rebalancing is also expensive. With a simple hash(key) mod N rule, changing N changes the target shard for most rows.

3. Consistent hashing

Consistent hashing is designed for systems where the set of nodes changes regularly. Instead of mapping keys directly with a simple modulo, it places both shards and keys onto a logical ring. A key belongs to the next shard clockwise on that ring.

The important property is limited movement during topology changes. When a shard is added or removed, only the keys near the affected segment need to move. That makes consistent hashing attractive for caches, object stores, and elastic distributed systems.

It still needs care. A small number of physical nodes on the ring can create uneven ownership, so many systems add virtual positions for each node. It also solves rebalancing better than plain hashing, not query-shape problems such as cross-shard joins or ordered scans.

4. Virtual bucket sharding

Virtual bucket sharding adds an indirection layer. Data is first assigned to many logical buckets, and those buckets are then mapped onto physical shards. If 1,024 buckets exist, a shard might own buckets 0 to 127 today and a different set next month.

This model is operationally attractive because bucket ownership can change without redefining the placement rule for every record. Rebalancing becomes a controlled remapping of selected buckets instead of a broad redistribution. It also supports gradual expansion because new machines can take over a subset of buckets first.

The cost is extra metadata and control-plane logic. The system must track bucket ownership reliably and route requests through that map. If the mapping becomes stale or inconsistent, requests can be misrouted even when the data itself is healthy.

There is no universal winner. Range sharding favours locality, hash sharding favours balance, consistent hashing favours elastic membership, and virtual buckets favour operational control. Choose the algorithm that matches your real pain point: hotspot avoidance, scan efficiency, rebalancing cost, or day-two operations.