Exactly Once Idempotent Consumer Store in Python

A mock key-value store that guarantees exactly-once processing by rejecting duplicate message keys in a message or event stream.

Easy Python 3.9+ Aug 9, 2026 Streaming & messaging 15 views 0 copies

idempotency streaming deduplication consumer

Python code

22 lines

Python 3.9+

from collections import defaultdict

class ExactlyOnceStore:
    def __init__(self):
        self.processed = defaultdict(set)
        self.data = {}

    def consume(self, key, value):
        if key in self.data:
            return False
        self.data[key] = value
        return True

    def get_processed_count(self):
        return len(self.data)

if __name__ == "__main__":
    store = ExactlyOnceStore()
    print(store.consume("user:1", {"name": "Alice"}))
    print(store.consume("user:1", {"name": "Alice Again"}))
    print(store.consume("user:2", {"name": "Bob"}))
    print("Processed count:", store.get_processed_count())

Output

stdout

True
False
True
Processed count: 2

How it works

The ExactlyOnceStore uses a plain dictionary as a deduplication key, so the first consume call for a key stores the value and returns True. Any later call with the same key is rejected with False because the key already exists. This simulates an idempotent consumer in a streaming system, where re-delivered events must not be processed twice. Using defaultdict(set) for processed is a common pattern, though in this version data itself is the deduplication source.

Common mistakes

Confusing `defaultdict(set)` as the dedup source instead of a real set of processed keys
Not resetting the store between test runs when verifying exactly-once behavior
Assuming the check must be thread-safe for a real distributed system — this mock is single-threaded only

Variations

Replace the dictionary with a Redis-backed store keyed on message ID for distributed dedup
Add a TTL or cleanup routine so old keys expire and memory doesn't grow forever

Real-world use cases

Deduplicating Kafka or SQS re-delivered events by their message ID so side effects don't run twice.
Guaranteeing a webhook or payment callback is applied only once despite retries.
Storing processed job identifiers in a lookup table while building an idempotent batch worker.

Exactly Once Idempotent Consumer Store in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Streaming & messaging

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Streaming & messaging

Keep learning

Tutorials

Quizzes