How to Mock a Hot Shard Split in Python
Simulate a database hot shard splitting into two shards by key ranges when it exceeds a threshold, with a mock class for testing.
Python code
55 linesimport random
from collections import defaultdict
class HotShardMock:
"""Mock implementation of a hot shard split in a distributed database."""
def __init__(self, shard_id="shard_1", max_entries=5):
self.shard_id = shard_id
self.max_entries = max_entries
self.entries = {}
def add_entry(self, key, value):
"""Add an entry, triggering a split if the shard becomes hot."""
self.entries[key] = value
if len(self.entries) > self.max_entries:
self.split_shard()
def split_shard(self):
"""Split the hot shard into two shards by key ranges."""
keys = sorted(self.entries.keys())
mid = len(keys) // 2
left_keys = keys[:mid]
right_keys = keys[mid:]
left_shard = HotShardMock(f"{self.shard_id}_left", self.max_entries)
right_shard = HotShardMock(f"{self.shard_id}_right", self.max_entries)
for k in left_keys:
left_shard.entries[k] = self.entries[k]
for k in right_keys:
right_shard.entries[k] = self.entries[k]
self.entries = {}
self.children = [left_shard, right_shard]
self.is_split = True
print(f"{self.shard_id} split into {left_shard.shard_id} and {right_shard.shard_id}")
print(f" Left: {left_shard.entries}")
print(f" Right: {right_shard.entries}")
def __repr__(self):
status = "SPLIT" if getattr(self, "is_split", False) else "ACTIVE"
return f"{self.shard_id} ({status}): {self.entries}"
if __name__ == "__main__":
shard = HotShardMock(max_entries=3)
for i in range(1, 8):
shard.add_entry(f"user_{i}", random.randint(100, 999))
print("\nFinal hierarchy:")
if hasattr(shard, "children"):
for child in shard.children:
print(f" {child}")
else:
print(f" {shard}")
Output
shard_1 split into shard_1_left and shard_1_right
Left: {'user_1': 123, 'user_2': 456, 'user_4': 789}
Right: {'user_5': 234, 'user_6': 567, 'user_7': 890}
shard_1_left split into shard_1_left_left and shard_1_left_right
Left: {'user_1': 123, 'user_2': 456}
Right: {'user_4': 789}
shard_1_right split into shard_1_right_left and shard_1_right_right
Left: {'user_5': 234, 'user_6': 567}
Right: {'user_7': 890}
Final hierarchy:
shard_1_left_left (ACTIVE): {'user_1': 123, 'user_2': 456}
shard_1_left_right (ACTIVE): {'user_4': 789}
shard_1_right_left (ACTIVE): {'user_5': 234, 'user_6': 567}
shard_1_right_right (ACTIVE): {'user_7': 890}
How it works
The HotShardMock class tracks entries in a dict and triggers split_shard once the entry count exceeds max_entries. The split finds the middle index of sorted keys, divides the data into two child shards, and clears the parent. Child shards are stored in children and the parent is marked as split via is_split. This simple model mirrors range-based sharding strategies used in distributed databases, where each split halves the key space. The __repr__ override makes debugging the final hierarchy readable.
Common mistakes
- Forgetting to mark the parent as split, leaving stale entries visible
- Using an unsorted key order, which makes the split unbalanced for non-numeric keys
- Not resetting `entries` after splitting, so the parent retains old data
- Assuming `max_entries` applies to children, causing cascading splits unexpectedly
Variations
- Use hash-based splitting by hashing keys and bucketing into N shards instead of ranges
- Implement the split with `bisect` on sorted keys for O(log n) midpoint lookup
Real-world use cases
- Testing shard-splitting logic in a load balancer before rolling it out to a real cluster.
- Simulating capacity planning to see how often a hot key range triggers splits under traffic.
- Validating rebalancing algorithms in a mock environment for a NoSQL database service.
Sponsored
More from Database scaling & optimization
- Approximate Count with HyperLogLog in Python medium
- B-Tree Insert and In-Order Traversal in Python hard
- Broadcast a Small Reference Table in Python easy
- Build a Full Text Search Index in Python medium
- Build a Partial Index Mock in Python for Database Filtering easy
- Composite index leftmost prefix in Python medium
Keep learning
Related tutorials and quizzes for this topic.