How to shard output by primary key hash mod N in Python

This code computes a consistent shard index for any primary key string using an MD5 hash mod the number of shards, enabling stable key-based data distribution.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 11 views 0 copies

hashing sharding hashlib data-distribution modulo

Python code

14 lines

Python 3.9+

import hashlib

def shard_id(primary_key: str, num_shards: int) -> int:
    """Return the shard index for a primary key using MD5 hash mod N."""
    digest = hashlib.md5(primary_key.encode("utf-8")).hexdigest()
    hash_int = int(digest, 16)
    return hash_int % num_shards

if __name__ == "__main__":
    keys = ["user_1001", "order_5002", "product_77", "session_abc"]
    num_shards = 5
    for key in keys:
        shard = shard_id(key, num_shards)
        print(f"{key} -> shard {shard}")

Output

stdout

user_1001 -> shard 2
order_5002 -> shard 4
product_77 -> shard 3
session_abc -> shard 0

How it works

The hashlib.md5 call converts the UTF-8 encoded key into a fixed-length 128-bit digest, then hexdigest() gives a 32-character hex string. Converting that hex string to an integer with int(digest, 16) preserves the full hash entropy. Taking the modulo with num_shards distributes keys roughly evenly across shards. Because MD5 is deterministic, the same key always maps to the same shard index — essential for consistent lookup and storage.

Common mistakes

Using Python's built-in `hash()` which is randomized per process and not stable across runs
Forgetting to encode the key to UTF-8 bytes before hashing
Using a string modulo instead of converting the digest to an integer first

Variations

Use SHA-256 instead of MD5 for better collision resistance: `hashlib.sha256(key.encode()).hexdigest()`
Use a keyed hash (HMAC) when shard assignment must not be guessable by clients

Real-world use cases

Distributing user records across multiple database partitions in a sharded PostgreSQL cluster.
Consistently routing incoming messages to one of several Kafka partitions by a message key.
Assigning files to a node in a distributed cache like Redis Cluster based on their cache key.

How to shard output by primary key hash mod N in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Keep learning

Tutorials

Quizzes