How to Build a Shard Map Mock Dict in Python

Implement a dictionary-like class that distributes keys across multiple shards using Python's hash() for realistic data partitioning.

Medium Python 3.9+ Aug 9, 2026 Database scaling & optimization 14 views 0 copies

Python code

45 lines
Python 3.9+
class ShardMap:
    def __init__(self, shard_count):
        self.shards = {i: {} for i in range(shard_count)}
        self.shard_count = shard_count

    def _shard_for(self, key):
        return hash(key) % self.shard_count

    def __getitem__(self, key):
        return self.shards[self._shard_for(key)][key]

    def __setitem__(self, key, value):
        self.shards[self._shard_for(key)][key] = value

    def __delitem__(self, key):
        del self.shards[self._shard_for(key)][key]

    def __contains__(self, key):
        shard = self.shards[self._shard_for(key)]
        return key in shard

    def __len__(self):
        return sum(len(shard) for shard in self.shards.values())

    def get(self, key, default=None):
        return self.shards[self._shard_for(key)].get(key, default)

    def keys(self):
        return [key for shard in self.shards.values() for key in shard]

    def __repr__(self):
        return f"ShardMap(shards={self.shards})"


if __name__ == "__main__":
    smap = ShardMap(shard_count=3)
    smap["apple"] = 10
    smap["banana"] = 20
    smap["cherry"] = 30

    print(smap["banana"])
    print(len(smap))
    print(smap.keys())
    print("apple" in smap)
    print(smap.shards)

Output

stdout
20
3
['banana', 'apple', 'cherry']
True
{0: {'banana': 20}, 1: {'cherry': 30}, 2: {'apple': 10}}

How it works

The ShardMap class mimics a distributed dictionary by partitioning keys into internal per-shard dicts. _shard_for uses hash(key) % shard_count to deterministically map each key to a shard, so lookups, inserts, and deletions only touch one shard. __getitem__, __setitem__, and __delitem__ delegate to the correct shard, while __contains__ and get provide safe access with defaults. __len__ aggregates counts from all shards, and keys() flattens every shard's keys. This pattern mirrors real sharded database clients where consistent key-to-shard routing is essential for scaling writes across nodes.

Common mistakes

  • Assuming hash() is stable across Python processes—it is randomized per run, so don't use it for persistent shard mapping.
  • Forgetting to keep the shard count fixed; changing `shard_count` would rehash keys and break data access.
  • Overriding `__repr__` to expose internal shards, which can leak sensitive data in production logs.
  • Not implementing `__iter__` or `items()`, so the mock dict isn't fully drop-in compatible with real dicts.

Variations

  1. Use a more stable hash like `hashlib.sha256(key.encode()).digest()[0]` for persistent shard mapping.
  2. Add `items()` and `__iter__` methods to make the class more dict-like and easier to iterate over.

Real-world use cases

  • Prototyping a sharded database client before implementing real wire protocol calls, letting you test key routing logic locally.
  • Modeling cache partitioning in a distributed system where each shard represents a separate Redis instance or memcached node.
  • Unit-testing sharding strategies by injecting a mock dict that simulates how keys are distributed across physical partitions.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Database scaling & optimization

Related tutorials and quizzes for this topic.