Partition Data by Hash Key Mod N in Python
Returns a partition index for a string key by hashing it with MD5 and taking modulo N, then groups sample keys into partitions.
Python code
20 linesimport hashlib
def partition_key(key: str, num_partitions: int) -> int:
"""Return partition index for key using MD5 hash mod N."""
digest = hashlib.md5(key.encode()).hexdigest()
return int(digest, 16) % num_partitions
if __name__ == "__main__":
keys = ["alice", "bob", "carol", "dave", "eve"]
num_partitions = 3
partitions = {}
for key in keys:
p = partition_key(key, num_partitions)
partitions.setdefault(p, []).append(key)
for p in sorted(partitions):
print(f"partition {p}: {partitions[p]}")Output
partition 0: ['dave', 'eve']
partition 1: ['alice', 'bob']
partition 2: ['carol']How it works
The function uses hashlib.md5 to create a deterministic hash of the key string. Converting the hex digest to an integer with int(digest, 16) gives a large number, and the modulo operation distributes keys evenly across partitions. Because MD5 is stable across runs, the same key always maps to the same partition, which is essential for consistent data partitioning. The example groups keys into a dictionary and prints each partition's members sorted by partition index.
Common mistakes
- Using Python's built-in `hash()` instead of a stable hash like MD5, which changes between runs.
- Forgetting to encode the string before hashing (`hashlib.md5(key.encode())`).
- Using a non-prime number of partitions can lead to uneven distribution if the hash is not uniform, but MD5 is uniform enough for most cases.
Variations
- Use a faster non-cryptographic hash like `hashlib.sha256` if you need longer digests, but MD5 is fine for partitioning.
- Partition key directly with `int(hashlib.md5(key.encode()).hexdigest(), 16) % num_partitions` in a single line.
Real-world use cases
- Sharding user data across multiple database shards by user ID for load balancing.
- Distributing events to different Kafka partitions to ensure order per key.
- Splitting a large dataset into parallel processing chunks in a Spark or MapReduce job.
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.