How to Remove Duplicates in Python Preserving Order

Removes duplicate items from a list while keeping the first occurrence order intact using a set for fast membership checks.

Easy Python 3.9+ Aug 9, 2026 Algorithms & data structures 14 views 0 copies

deduplication set list algorithms order-preserving

Python code

14 lines

Python 3.9+

def remove_duplicates_preserving_order(items):
    seen = set()
    result = []
    for item in items:
        if item not in seen:
            seen.add(item)
            result.append(item)
    return result

if __name__ == "__main__":
    sample = [3, 1, 2, 1, 3, 4, 2, 5]
    unique_items = remove_duplicates_preserving_order(sample)
    print(f"Original: {sample}")
    print(f"Unique (order preserved): {unique_items}")

Output

stdout

Original: [3, 1, 2, 1, 3, 4, 2, 5]
Unique (order preserved): [3, 1, 2, 4, 5]

How it works

The function uses a set named seen to track items already encountered. The if item not in seen check runs in O(1) on average because sets use hashing. Only items not yet in the set are appended to the result list, preserving their original order. This is more efficient than using list.count() or nested loops, which would be O(n²). The function works with any hashable items — integers, strings, tuples — making it widely reusable.

Common mistakes

Using `set(items)` which loses original order instead of preserving it
Forgetting that the input must contain hashable items (e.g., lists inside would fail)
Assuming the `seen` set mutation affects the original list — it doesn't

Variations

Use `dict.fromkeys(items)` to remove duplicates in Python 3.7+ while keeping order
Use a one-liner: `list(dict.fromkeys(items))` for a concise alternative

Real-world use cases

Deduplicate user IDs in an API response while keeping the sequence of events.
Clean up log entries where repeated messages should appear only at first occurrence.
Preserve the insertion order in a dropdown or config list scraped from a file.

How to Remove Duplicates in Python Preserving Order

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Algorithms & data structures

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Algorithms & data structures

Keep learning

Tutorials

Quizzes