Exactly Once Processing Dedupe Mock in Python

Implements a streaming deduplicator using a set and queue to guarantee each item is processed exactly once while preserving insertion order.

Easy Python 3.9+ Aug 9, 2026 Reliability & rate limiting 15 views 0 copies

deduplication exactly-once streaming queues sets

Python code

23 lines

Python 3.9+

from collections import deque

class DedupeStream:
    def __init__(self):
        self.seen = set()
        self.queue = deque()

    def add(self, item):
        if item not in self.seen:
            self.seen.add(item)
            self.queue.append(item)
            print(f"Processed: {item} (exactly once)")
        else:
            print(f"Deduped: {item}")

    def results(self):
        return list(self.queue)

if __name__ == "__main__":
    ds = DedupeStream()
    for item in ["apple", "banana", "apple", "cherry", "banana", "date"]:
        ds.add(item)
    print("Final exactly-once items:", ds.results())

Output

stdout

Processed: apple (exactly once)
Processed: banana (exactly once)
Deduped: apple
Processed: cherry (exactly once)
Deduped: banana
Processed: date (exactly once)
Final exactly-once items: ['apple', 'banana', 'cherry', 'date']

How it works

The DedupeStream class maintains a seen set for O(1) membership checks and a deque to preserve the insertion order of unique items. When add is called, it checks whether the item is already in seen; if not, it adds it to both structures and logs processing; otherwise it logs a dedupe event. The results method returns the list of items that were processed exactly once. This pattern is a simple mock for exactly-once semantics in message processing systems.

Common mistakes

Forgetting to add the item to both the set and the queue when it's new
Using a list for the seen check, leading to O(n) lookups instead of O(1)

Variations

Use `dict.fromkeys` to deduplicate a list while preserving order
Use a `hashlib` hash of the item to support larger objects

Real-world use cases

Deduplicating incoming webhook events from a message queue to avoid duplicate processing.
Ensuring each user action in an analytics pipeline is counted only once for billing.
Filtering duplicate IDs from a batch processing job before writing to a database.

Exactly Once Processing Dedupe Mock in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Reliability & rate limiting

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Reliability & rate limiting

Keep learning

Tutorials

Quizzes