How to Build a Shard Map Mock Dict in Python
Implement a dictionary-like class that distributes keys across multiple shards using Python's hash() for realistic data partitioning.
Python code
45 linesclass ShardMap:
def __init__(self, shard_count):
self.shards = {i: {} for i in range(shard_count)}
self.shard_count = shard_count
def _shard_for(self, key):
return hash(key) % self.shard_count
def __getitem__(self, key):
return self.shards[self._shard_for(key)][key]
def __setitem__(self, key, value):
self.shards[self._shard_for(key)][key] = value
def __delitem__(self, key):
del self.shards[self._shard_for(key)][key]
def __contains__(self, key):
shard = self.shards[self._shard_for(key)]
return key in shard
def __len__(self):
return sum(len(shard) for shard in self.shards.values())
def get(self, key, default=None):
return self.shards[self._shard_for(key)].get(key, default)
def keys(self):
return [key for shard in self.shards.values() for key in shard]
def __repr__(self):
return f"ShardMap(shards={self.shards})"
if __name__ == "__main__":
smap = ShardMap(shard_count=3)
smap["apple"] = 10
smap["banana"] = 20
smap["cherry"] = 30
print(smap["banana"])
print(len(smap))
print(smap.keys())
print("apple" in smap)
print(smap.shards)
Output
20
3
['banana', 'apple', 'cherry']
True
{0: {'banana': 20}, 1: {'cherry': 30}, 2: {'apple': 10}}
How it works
The ShardMap class mimics a distributed dictionary by partitioning keys into internal per-shard dicts. _shard_for uses hash(key) % shard_count to deterministically map each key to a shard, so lookups, inserts, and deletions only touch one shard. __getitem__, __setitem__, and __delitem__ delegate to the correct shard, while __contains__ and get provide safe access with defaults. __len__ aggregates counts from all shards, and keys() flattens every shard's keys. This pattern mirrors real sharded database clients where consistent key-to-shard routing is essential for scaling writes across nodes.
Common mistakes
- Assuming hash() is stable across Python processes—it is randomized per run, so don't use it for persistent shard mapping.
- Forgetting to keep the shard count fixed; changing `shard_count` would rehash keys and break data access.
- Overriding `__repr__` to expose internal shards, which can leak sensitive data in production logs.
- Not implementing `__iter__` or `items()`, so the mock dict isn't fully drop-in compatible with real dicts.
Variations
- Use a more stable hash like `hashlib.sha256(key.encode()).digest()[0]` for persistent shard mapping.
- Add `items()` and `__iter__` methods to make the class more dict-like and easier to iterate over.
Real-world use cases
- Prototyping a sharded database client before implementing real wire protocol calls, letting you test key routing logic locally.
- Modeling cache partitioning in a distributed system where each shard represents a separate Redis instance or memcached node.
- Unit-testing sharding strategies by injecting a mock dict that simulates how keys are distributed across physical partitions.
Sponsored
More from Database scaling & optimization
- Approximate Count with HyperLogLog in Python medium
- B-Tree Insert and In-Order Traversal in Python hard
- Broadcast a Small Reference Table in Python easy
- Build a Full Text Search Index in Python medium
- Build a Partial Index Mock in Python for Database Filtering easy
- Composite index leftmost prefix in Python medium
Keep learning
Related tutorials and quizzes for this topic.