Dedupe processed message IDs in Python

Filters an inbox of messages by removing items whose IDs have already been processed, using a set for fast lookups.

Easy Python 3.9+ Aug 9, 2026 Streaming & messaging 12 views 0 copies

deduplication streaming json set

Python code

22 lines

Python 3.9+

from pathlib import Path
import json


def dedupe_processed_ids(inbox_file: Path, processed_file: Path) -> list:
    processed = set(json.loads(processed_file.read_text()))
    inbox = json.loads(inbox_file.read_text())
    deduped = [item for item in inbox if item["id"] not in processed]
    return deduped


if __name__ == "__main__":
    inbox_data = [{"id": 1, "msg": "hello"}, {"id": 2, "msg": "world"}, {"id": 3, "msg": "again"}]
    processed_ids = [2]
    inbox_file = Path("inbox.json")
    processed_file = Path("processed.json")
    inbox_file.write_text(json.dumps(inbox_data))
    processed_file.write_text(json.dumps(processed_ids))
    result = dedupe_processed_ids(inbox_file, processed_file)
    print(result)
    inbox_file.unlink()
    processed_file.unlink()

Output

stdout

[{'id': 1, 'msg': 'hello'}, {'id': 3, 'msg': 'again'}]

How it works

The function reads both files as JSON. Processed IDs are loaded into a set for O(1) membership tests. A list comprehension keeps only inbox items whose id is not in the processed set. Using a set avoids duplicate IDs and makes deduplication efficient even with large lists. The files are written and then removed to demonstrate the use case without leaving artifacts.

Common mistakes

Forgetting to convert processed list to a set, causing slower lookups.
Assuming message IDs are unique within the inbox, which may cause unexpected removals.
Not handling missing keys if the JSON structure varies.

Variations

Use a dict comprehension to keep original order while filtering.
Read files using `json.load` with an open file handle for memory efficiency.

Real-world use cases

Deduplicating events pulled from a message queue before processing them in a worker.
Skipping already-handled webhook deliveries when re-delivery can happen.
Filtering out previously seen notifications or alerts in a monitoring system.

Dedupe processed message IDs in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Streaming & messaging

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Streaming & messaging

Keep learning

Tutorials

Quizzes