How to Deduplicate Events with At-Least-Once Delivery in Python

Implements an exactly-once processing pattern for at-least-once event delivery by tracking seen event IDs in a set, skipping duplicates.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 13 views 0 copies

deduplication idempotent event-processing

Python code

20 lines

Python 3.9+

seen_ids = set()

def process_event(event_id: str, payload: dict) -> dict:
    """Process an event exactly once, ignoring duplicates."""
    if event_id in seen_ids:
        return {"status": "duplicate", "event_id": event_id}
    seen_ids.add(event_id)
    return {"status": "processed", "event_id": event_id, **payload}

if __name__ == "__main__":
    events = [
        ("e1", {"amount": 100}),
        ("e2", {"amount": 200}),
        ("e1", {"amount": 100}),  # duplicate
        ("e3", {"amount": 300}),
        ("e2", {"amount": 200}),  # duplicate
    ]
    for event_id, payload in events:
        result = process_event(event_id, payload)
        print(result)

Output

stdout

{'status': 'processed', 'event_id': 'e1', 'amount': 100}
{'status': 'processed', 'event_id': 'e2', 'amount': 200}
{'status': 'duplicate', 'event_id': 'e1'}
{'status': 'processed', 'event_id': 'e3', 'amount': 300}
{'status': 'duplicate', 'event_id': 'e2'}

How it works

The function uses a global set seen_ids to record every event ID that has been processed. On each call, it checks if the event ID is already in the set; if so, it returns a duplicate status without reprocessing. Otherwise, it adds the ID to the set and processes the event, ensuring exactly-once side effects despite at-least-once delivery. This pattern is essential for idempotent downstream systems. In production, the set would be backed by a persistent store like Redis or a database to survive restarts.

Common mistakes

Using a list instead of a set, causing O(n) lookup and duplicate storage
Forgetting to make the dedupe store persistent across restarts
Assuming event IDs are unique across systems without namespacing

Variations

Use Redis `SETNX` or a database unique constraint for distributed dedupe
Wrap processing in a transaction with a conditional insert for atomicity

Real-world use cases

Message queue consumers that must process each event exactly once even with retries
Webhook handlers that receive duplicate notifications and must ignore repeats
Data pipeline stages where upstream jobs may re-emit the same record

How to Deduplicate Events with At-Least-Once Delivery in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Keep learning

Tutorials

Quizzes