Implementing Data Partitioning

1. Understanding Partitioning Strategies

StrategyMechanismTrade-offs
Hashpartition = hash(key) mod NEven distribution; bad for range queries
RangeContiguous key ranges per partitionRange scans efficient; hot ranges possible
Consistent HashHash ring with virtual nodesMinimal data movement on rebalance
Directory / LookupExternal metadata maps key → nodeFlexible; lookup overhead, SPOF
CompositeHash(tenant) + range(time)Combines benefits

2. Implementing Consistent Hashing

Example: Consistent hash ring with virtual nodes

public class ConsistentHashRing<T> {
    private final SortedMap<Long, T> ring = new TreeMap<>();
    private final int vnodesPerNode;

    public ConsistentHashRing(int vnodesPerNode) { this.vnodesPerNode = vnodesPerNode; }

    public void addNode(T node) {
        for (int i = 0; i < vnodesPerNode; i++) {
            ring.put(hash(node + "#" + i), node);
        }
    }

    public T getNode(String key) {
        if (ring.isEmpty()) return null;
        long h = hash(key);
        SortedMap<Long, T> tail = ring.tailMap(h);
        Long target = tail.isEmpty() ? ring.firstKey() : tail.firstKey();
        return ring.get(target);
    }

    private long hash(String s) {
        return Hashing.murmur3_128().hashString(s, StandardCharsets.UTF_8).asLong();
    }
}
PropertyValue
Data movement on add/removeK/N keys (vs K with mod N)
Recommended vnodes100-256 per physical node
Hash functionMurmurHash3, xxHash, SHA-1

3. Implementing Virtual Nodes (vnodes)

BenefitDetail
Load balanceSmooths skew across physical nodes
Heterogeneous capacityLarger nodes get more vnodes
Faster rebalanceMany small transfers, parallel
Used byCassandra, DynamoDB, Riak

4. Implementing Hash Slots (Redis Cluster)

PropertyValue
Total slots16384 (2¹⁴)
Slot computationCRC16(key) mod 16384
Hash tags{user:42}.profile — only braces hashed for co-location
ReshardingSlot migration via MIGRATE

5. Implementing Range Partitioning

AspectDetail
LayoutSorted key ranges per partition (e.g., [a-f], [g-m])
StrengthRange scans, ordered iteration
RiskHot range (sequential keys → one shard)
Auto-splitSplitter divides hot range into two
Used byHBase, BigTable, Spanner, CockroachDB

6. Handling Partition Rebalancing

StrategyMechanismPros / Cons
Hash mod NReassign all keysSimple; massive movement
Fixed partitions10× nodes; reassign whole partitionsPredictable; coarse
Dynamic splitSplit when partition exceeds sizeAdapts to growth (HBase)
Consistent hashMove only affected vnodesMinimal movement
Warning: Avoid automatic rebalancing during peak traffic — schedule or rate-limit transfers.

7. Understanding Hot Spots and Skewed Workloads

CauseMitigation
Celebrity user / tenantSalt key (userId#shardN) and fan-in on read
Sequential timestampsHash prefix the time key
Single trending entityCache + replicate hot keys
Skewed rangeAuto-split hot range

8. Selecting Partition Keys

CriterionWhy
High cardinalitySpreads load
Even access patternAvoids hot shards
Aligns with queryReduces cross-partition scatter-gather
StableAvoids re-sharding

9. Handling Cross-Partition Queries

PatternApproach
Scatter-gatherFan out to all partitions, merge
Parallel aggregationMap-reduce style
Materialized viewPre-compute by alternate key
Global secondary indexSeparate partitioning by index key

10. Implementing Secondary Indexes in Partitioned Systems

TypeLayoutTrade-offs
Local (document-partitioned)Index lives with data partitionCheap writes; scatter-gather reads
Global (term-partitioned)Index partitioned by index keyFast reads; distributed writes
ExamplesDynamoDB GSI vs LSI; ES per-shard inverted index