Idempotent Pipeline Dedupe by Record ID Set in Python

Filters records against a persistent set of seen IDs, returning only new ones and the updated set for idempotent pipeline processing.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 12 views 0 copies

deduplication idempotency pipelines sets data-processing

Python code

30 lines

Python 3.9+

def dedupe_records(records, seen_ids=None):
    """Return records whose id has not been seen before."""
    if seen_ids is None:
        seen_ids = set()
    unique = []
    for record in records:
        record_id = record.get("id")
        if record_id not in seen_ids:
            seen_ids.add(record_id)
            unique.append(record)
    return unique, seen_ids


if __name__ == "__main__":
    batch1 = [
        {"id": 1, "name": "Alice"},
        {"id": 2, "name": "Bob"},
    ]
    batch2 = [
        {"id": 2, "name": "Bob"},
        {"id": 3, "name": "Charlie"},
    ]

    seen = set()
    result1, seen = dedupe_records(batch1, seen)
    result2, seen = dedupe_records(batch2, seen)

    print("After batch1:", result1)
    print("After batch2:", result2)
    print("Total seen ids:", sorted(seen))

Output

stdout

After batch1: [{'id': 1, 'name': 'Alice'}, {'id': 2, 'name': 'Bob'}]
After batch2: [{'id': 3, 'name': 'Charlie'}]
Total seen ids: [1, 2, 3]

How it works

The seen_ids set is passed by reference and mutated inside the function, so the caller sees accumulated IDs across batches. Using a set for membership checks is O(1) on average, making this efficient for large volumes. The optional seen_ids parameter defaults to None to avoid the mutable default argument pitfall; a fresh set is created on first call. The function returns both the unique records and the updated set, which lets you carry the state forward for true idempotency.

Common mistakes

Using a mutable default like `seen_ids=set()` instead of None - this persists state across function calls unintentionally.
Forgetting that `.get('id')` returns None for missing keys, which would dedupe all missing-ID records incorrectly.
Assuming records are sorted - the function preserves original order but does not guarantee id uniqueness within the same batch beyond the set logic.

Variations

Use a generator expression to yield unique records lazily for streaming scenarios.
Instead of building a list, yield records with `yield` and update the set, reducing memory footprint for huge feeds.

Real-world use cases

Deduplicating events from a message queue before inserting into a database, ensuring at-least-once delivery doesn't cause duplicates.
Loading batches of CSV rows where each row has a business key, skipping already-processed IDs across multiple file chunks.
Ingesting API webhook payloads with a persistent dedupe set to prevent reprocessing the same transaction twice.

Idempotent Pipeline Dedupe by Record ID Set in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Keep learning

Tutorials

Quizzes