How to Mock a Hot Shard Split in Python

Simulate a database hot shard splitting into two shards by key ranges when it exceeds a threshold, with a mock class for testing.

Medium Python 3.9+ Aug 9, 2026 Database scaling & optimization 16 views 0 copies

Python code

55 lines
Python 3.9+
import random
from collections import defaultdict


class HotShardMock:
    """Mock implementation of a hot shard split in a distributed database."""

    def __init__(self, shard_id="shard_1", max_entries=5):
        self.shard_id = shard_id
        self.max_entries = max_entries
        self.entries = {}

    def add_entry(self, key, value):
        """Add an entry, triggering a split if the shard becomes hot."""
        self.entries[key] = value
        if len(self.entries) > self.max_entries:
            self.split_shard()

    def split_shard(self):
        """Split the hot shard into two shards by key ranges."""
        keys = sorted(self.entries.keys())
        mid = len(keys) // 2
        left_keys = keys[:mid]
        right_keys = keys[mid:]

        left_shard = HotShardMock(f"{self.shard_id}_left", self.max_entries)
        right_shard = HotShardMock(f"{self.shard_id}_right", self.max_entries)

        for k in left_keys:
            left_shard.entries[k] = self.entries[k]
        for k in right_keys:
            right_shard.entries[k] = self.entries[k]

        self.entries = {}
        self.children = [left_shard, right_shard]
        self.is_split = True
        print(f"{self.shard_id} split into {left_shard.shard_id} and {right_shard.shard_id}")
        print(f"  Left:  {left_shard.entries}")
        print(f"  Right: {right_shard.entries}")

    def __repr__(self):
        status = "SPLIT" if getattr(self, "is_split", False) else "ACTIVE"
        return f"{self.shard_id} ({status}): {self.entries}"


if __name__ == "__main__":
    shard = HotShardMock(max_entries=3)
    for i in range(1, 8):
        shard.add_entry(f"user_{i}", random.randint(100, 999))
    print("\nFinal hierarchy:")
    if hasattr(shard, "children"):
        for child in shard.children:
            print(f"  {child}")
    else:
        print(f"  {shard}")

Output

stdout
shard_1 split into shard_1_left and shard_1_right
  Left:  {'user_1': 123, 'user_2': 456, 'user_4': 789}
  Right: {'user_5': 234, 'user_6': 567, 'user_7': 890}
shard_1_left split into shard_1_left_left and shard_1_left_right
  Left:  {'user_1': 123, 'user_2': 456}
  Right: {'user_4': 789}
shard_1_right split into shard_1_right_left and shard_1_right_right
  Left:  {'user_5': 234, 'user_6': 567}
  Right: {'user_7': 890}

Final hierarchy:
  shard_1_left_left (ACTIVE): {'user_1': 123, 'user_2': 456}
  shard_1_left_right (ACTIVE): {'user_4': 789}
  shard_1_right_left (ACTIVE): {'user_5': 234, 'user_6': 567}
  shard_1_right_right (ACTIVE): {'user_7': 890}

How it works

The HotShardMock class tracks entries in a dict and triggers split_shard once the entry count exceeds max_entries. The split finds the middle index of sorted keys, divides the data into two child shards, and clears the parent. Child shards are stored in children and the parent is marked as split via is_split. This simple model mirrors range-based sharding strategies used in distributed databases, where each split halves the key space. The __repr__ override makes debugging the final hierarchy readable.

Common mistakes

  • Forgetting to mark the parent as split, leaving stale entries visible
  • Using an unsorted key order, which makes the split unbalanced for non-numeric keys
  • Not resetting `entries` after splitting, so the parent retains old data
  • Assuming `max_entries` applies to children, causing cascading splits unexpectedly

Variations

  1. Use hash-based splitting by hashing keys and bucketing into N shards instead of ranges
  2. Implement the split with `bisect` on sorted keys for O(log n) midpoint lookup

Real-world use cases

  • Testing shard-splitting logic in a load balancer before rolling it out to a real cluster.
  • Simulating capacity planning to see how often a hot key range triggers splits under traffic.
  • Validating rebalancing algorithms in a mock environment for a NoSQL database service.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Database scaling & optimization

Related tutorials and quizzes for this topic.