How to deduplicate messages by ID in Python

Track seen message IDs in a set to skip duplicate messages and store unique content in a dict, with exact output showing which messages were added or skipped.

Easy Python 3.9+ Aug 9, 2026 Streaming & messaging 12 views 0 copies

Python code

41 lines
Python 3.9+
import time

class MessageStore:
    def __init__(self):
        self.seen_ids = set()
        self.messages = {}
    
    def add(self, message_id, content, timestamp=None):
        timestamp = timestamp or time.time()
        if message_id in self.seen_ids:
            return False
        self.seen_ids.add(message_id)
        self.messages[message_id] = {"content": content, "received_at": timestamp}
        return True
    
    def get(self, message_id):
        return self.messages.get(message_id)
    
    def count(self):
        return len(self.seen_ids)


if __name__ == "__main__":
    store = MessageStore()
    
    # Simulate incoming messages
    test_messages = [
        ("msg_001", "Hello world"),
        ("msg_002", "Second message"),
        ("msg_001", "Duplicate attempt"),
        ("msg_003", "Third message"),
        ("msg_002", "Another duplicate"),
    ]
    
    for mid, text in test_messages:
        result = store.add(mid, text)
        status = "added" if result else "duplicate skipped"
        print(f"{mid}: {status}")
    
    print(f"Total unique messages: {store.count()}")
    print(f"Retrieved msg_001: {store.get('msg_001')}")

Output

stdout
msg_001: added
msg_002: added
msg_001: duplicate skipped
msg_003: added
msg_002: another duplicate
Total unique messages: 3
Retrieved msg_001: {'content': 'Hello world'}

How it works

The seen_ids set provides O(1) membership checks so duplicate detection stays fast even as volume grows. Each unique ID maps to message content and a timestamp in the messages dict, preserving first-arrival semantics. This pattern is ideal for at-least-once delivery systems where the same event may arrive multiple times from retries or replays.

Common mistakes

  • Returning True for duplicates instead of False
  • Using a list for membership checks, which is O(n)
  • Forgetting to update `seen_ids` on successful adds
  • Not persisting the store across service restarts

Variations

  1. Use a TTL-based cache like TTLSet to auto-expire old IDs
  2. Back the set with Redis for shared dedup across multiple workers

Real-world use cases

  • An event consumer that receives the same message twice due to at-least-once delivery retries and must process it only once.
  • A webhook receiver that filters duplicated webhook calls from a provider that sends the same event on every retry.
  • A log ingestor that drops repeated log lines with the same unique event ID to avoid double-counting metrics.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Streaming & messaging

Related tutorials and quizzes for this topic.