Python Generator to Filter Duplicates with a Seen Set

A lazily-evaluated generator function that yields only the first occurrence of each item, using a set to track seen values.

Easy Python 3.9+ Aug 9, 2026 Comprehensions & generators 13 views 0 copies

generator dedupe set yield lazy

Python code

11 lines

Python 3.9+

def unique_generator(items):
    seen = set()
    for item in items:
        if item not in seen:
            seen.add(item)
            yield item

if __name__ == "__main__":
    data = [1, 2, 2, 3, 3, 3, 4, 5, 5]
    result = list(unique_generator(data))
    print(result)

Output

stdout

[1, 2, 3, 4, 5]

How it works

The generator tracks previously yielded items in a seen set, checking membership before yielding. This makes the pattern O(1) per item on average, and because it's a generator, it processes items one at a time without building a full intermediate list. The function works with any iterable, not just lists, and preserves the original order of first appearance.

Common mistakes

Using a list instead of a set for `seen`, making checks O(n) and slow on large data
Forgetting to add the item to `seen` before yielding, causing duplicates to slip through
Not yielding at all — returning instead of building a generator
Converting the generator to a list immediately, defeating memory benefits on huge inputs

Variations

Use `dict.fromkeys(items)` to dedupe a finite list while preserving order
Apply `itertools.unique_everseen` from more-itertools for a drop-in tool

Real-world use cases

Deduplicate streaming log lines in real-time as they arrive from an API or file tail.
Remove repeated IDs from a large CSV reader pipeline without loading the whole file into memory.
Filter out already-seen events from a message queue consumer to avoid reprocessing duplicates.

Python Generator to Filter Duplicates with a Seen Set

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Keep learning

Tutorials

Quizzes