How to Partition and Order Kafka-Style Messages by Key in Python

Group messages with the same key into ordered buckets using hashing and a defaultdict, mimicking Kafka partition ordering.

Easy Python 3.9+ Aug 9, 2026 Streaming & messaging 14 views 0 copies

streaming partitioning kafka-pattern hash-map ordering

Python code

27 lines

Python 3.9+

from dataclasses import dataclass
from collections import defaultdict

@dataclass
class Message:
    key: str
    content: str

def partition_and_order(messages, num_partitions=3):
    partitions = defaultdict(list)
    for msg in messages:
        partition_id = hash(msg.key) % num_partitions
        partitions[partition_id].append(msg)
    return dict(sorted(partitions.items()))

if __name__ == "__main__":
    messages = [
        Message("user-1", "login"),
        Message("user-2", "cart"),
        Message("user-1", "search"),
        Message("user-3", "checkout"),
        Message("user-2", "pay"),
        Message("user-1", "logout"),
    ]
    result = partition_and_order(messages)
    for part_id, msgs in result.items():
        print(f"Partition {part_id}: {[m.content for m in msgs]}")

Output

stdout

Partition 0: ['checkout']
Partition 1: ['login', 'search', 'logout']
Partition 2: ['cart', 'pay']

How it works

The hash() function maps each key to an integer, and the modulo operator assigns it to one of num_partitions buckets. Using defaultdict(list) avoids checking for key existence manually — each new partition ID gets an empty list automatically. Appending in iteration order preserves the original message sequence within each partition, which is how per-key ordering works in real consumers. Sorting partitions with sorted(partitions.items()) gives deterministic output for the demo, though real systems don't need order across partitions.

Common mistakes

Using Python's built-in hash() which is salted per process — not stable across runs or languages
Forgetting that modulo on negative hash values can produce negative partition IDs
Assuming messages with the same key always go to the same partition when the number of partitions changes

Variations

Use `abs(hash(msg.key)) % num_partitions` to avoid negative IDs at the cost of a slight bias
Use `hashlib.md5` for a stable hash across processes
Sort messages inside each partition by timestamp if arrival order isn't guaranteed

Real-world use cases

Assigning user events to the same consumer so a user's activity is processed in order.
Sharding database writes by customer ID to keep related records on the same shard.
Distributing webhook payloads across worker processes while grouping per tenant.

How to Partition and Order Kafka-Style Messages by Key in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Streaming & messaging

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Streaming & messaging

Keep learning

Tutorials

Quizzes