How to deduplicate messages by ID in Python
Track seen message IDs in a set to skip duplicate messages and store unique content in a dict, with exact output showing which messages were added or skipped.
Python code
41 linesimport time
class MessageStore:
def __init__(self):
self.seen_ids = set()
self.messages = {}
def add(self, message_id, content, timestamp=None):
timestamp = timestamp or time.time()
if message_id in self.seen_ids:
return False
self.seen_ids.add(message_id)
self.messages[message_id] = {"content": content, "received_at": timestamp}
return True
def get(self, message_id):
return self.messages.get(message_id)
def count(self):
return len(self.seen_ids)
if __name__ == "__main__":
store = MessageStore()
# Simulate incoming messages
test_messages = [
("msg_001", "Hello world"),
("msg_002", "Second message"),
("msg_001", "Duplicate attempt"),
("msg_003", "Third message"),
("msg_002", "Another duplicate"),
]
for mid, text in test_messages:
result = store.add(mid, text)
status = "added" if result else "duplicate skipped"
print(f"{mid}: {status}")
print(f"Total unique messages: {store.count()}")
print(f"Retrieved msg_001: {store.get('msg_001')}")
Output
msg_001: added
msg_002: added
msg_001: duplicate skipped
msg_003: added
msg_002: another duplicate
Total unique messages: 3
Retrieved msg_001: {'content': 'Hello world'}
How it works
The seen_ids set provides O(1) membership checks so duplicate detection stays fast even as volume grows. Each unique ID maps to message content and a timestamp in the messages dict, preserving first-arrival semantics. This pattern is ideal for at-least-once delivery systems where the same event may arrive multiple times from retries or replays.
Common mistakes
- Returning True for duplicates instead of False
- Using a list for membership checks, which is O(n)
- Forgetting to update `seen_ids` on successful adds
- Not persisting the store across service restarts
Variations
- Use a TTL-based cache like TTLSet to auto-expire old IDs
- Back the set with Redis for shared dedup across multiple workers
Real-world use cases
- An event consumer that receives the same message twice due to at-least-once delivery retries and must process it only once.
- A webhook receiver that filters duplicated webhook calls from a provider that sends the same event on every retry.
- A log ingestor that drops repeated log lines with the same unique event ID to avoid double-counting metrics.
Sponsored
More from Streaming & messaging
- At Most Once Fire-and-Forget Mock in Python easy
- Batch Consume Process Commit Pattern in Python medium
- Build a Streaming Messaging Helper in Python easy
- Dead Letter Queue Failed Messages List Mock in Python easy
- Dedupe processed message IDs in Python easy
- Event Envelope with Schema Version Field in Python easy
Keep learning
Related tutorials and quizzes for this topic.