How to Deduplicate Messages in Python by ID

This code consumes a mock inbox of JSON messages and deduplicates them by message ID, keeping either the first or last occurrence.

Easy Python 3.9+ Aug 9, 2026 Reliability & rate limiting 15 views 0 copies

deduplication inbox json ordereddict

Python code

25 lines

Python 3.9+

import json
from collections import OrderedDict

mock_inbox = [
    {"id": 1, "message": "hello", "timestamp": "2024-01-01T10:00:00Z"},
    {"id": 2, "message": "world", "timestamp": "2024-01-01T10:01:00Z"},
    {"id": 1, "message": "hello", "timestamp": "2024-01-01T10:00:00Z"},
    {"id": 3, "message": "test", "timestamp": "2024-01-01T10:02:00Z"},
    {"id": 2, "message": "world", "timestamp": "2024-01-01T10:01:00Z"},
]


def consume_with_dedupe(inbox, keep="last"):
    seen = OrderedDict()
    for item in inbox:
        if keep == "last":
            seen[item["id"]] = item
        else:
            seen.setdefault(item["id"], item)
    return list(seen.values())


if __name__ == "__main__":
    deduped = consume_with_dedupe(mock_inbox)
    print(json.dumps(deduped, indent=2))

Output

stdout

[
  {
    "id": 1,
    "message": "hello",
    "timestamp": "2024-01-01T10:00:00Z"
  },
  {
    "id": 2,
    "message": "world",
    "timestamp": "2024-01-01T10:01:00Z"
  },
  {
    "id": 3,
    "message": "test",
    "timestamp": "2024-01-01T10:02:00Z"
  }
]

How it works

The consume_with_dedupe function uses an OrderedDict to track seen message IDs while preserving insertion order. For keep="last", it overwrites the value for an existing ID, so the last occurrence wins but the original position is kept. For keep="first", it uses setdefault to only add an item if the ID hasn't been seen. Converting the dictionary's values back to a list yields the deduplicated messages in correct order.

Common mistakes

Assuming IDs are unique without checking duplicates in real data
Forgetting that `OrderedDict` preserves insertion order but overwrites values in place
Not handling missing 'id' keys with .get() which can raise KeyError

Variations

Use a plain `dict` (Python 3.7+ guarantees order) with `seen.setdefault(id, item)` for first occurrence
Use `itertools.groupby` after sorting by ID to group consecutive duplicates

Real-world use cases

Consuming messages from a queue where duplicates can occur due to retries or network issues.
Deduplicating webhook events based on an event ID before processing them in a worker.
Cleaning up logs or audit trails to avoid duplicate entries from re-sent records.

How to Deduplicate Messages in Python by ID

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Reliability & rate limiting

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Reliability & rate limiting

Keep learning

Tutorials

Quizzes