How to Remove Duplicates from a Python List: Easy Methods

If you've been working with Python for any length of time, you've probably run into a situation where your list has more copies of the same item than you'd like. Maybe you're cleaning up user input, processing data from an API, or just trying to make your output look cleaner. Whatever the reason, removing duplicates from a list is one of those tasks that comes up again and again.

Let me walk you through the most practical ways to handle this, from the simplest one-liner to methods that preserve the original order of your items.

The Quickest Way: Using set()

The most straightforward approach is converting your list to a set. Sets in Python only hold unique items, so duplicates vanish automatically.

original_list = [3, 5, 2, 3, 8, 5, 1, 2]
unique_list = list(set(original_list))
print(unique_list)  # Output: [1, 2, 3, 5, 8]

This method is incredibly fast, especially with large lists. But there's a catch — sets don't maintain the original order of elements. If you don't care about order, this is your best bet. At PythonSkillset, we often use this approach when processing log files where the sequence doesn't matter.

Preserving Order with a Simple Loop

When order matters, you need a different approach. Here's a clean method that keeps things in their original sequence:

def remove_duplicates_ordered(items):
    seen = set()
    result = []
    for item in items:
        if item not in seen:
            seen.add(item)
            result.append(item)
    return result

data = [4, 2, 4, 1, 3, 2, 1]
clean_data = remove_duplicates_ordered(data)
print(clean_data)  # Output: [4, 2, 1, 3]

This works by tracking what we've already encountered. Each new unique item gets added to both the seen set and the result list. The set gives us O(1) lookup time, so this stays efficient even with thousands of items.

Using Dictionary Keys (Python 3.7+)

Since Python 3.7, dictionaries preserve insertion order. This gives us a neat trick:

data = [7, 2, 7, 4, 2, 9, 4]
unique = list(dict.fromkeys(data))
print(unique)  # Output: [7, 2, 4, 9]

The dict.fromkeys() method creates a dictionary where each list item becomes a key. Since dictionary keys must be unique, duplicates are automatically dropped. And because modern Python dictionaries remember insertion order, you get back a list with the original sequence preserved.

This is probably my go-to method at PythonSkillset when I need both uniqueness and order. It's concise and readable.

Handling Lists of Dictionaries or Custom Objects

What if your list contains dictionaries or other unhashable types? Sets won't work directly because dictionaries aren't hashable. Here's a practical workaround:

users = [
    {"name": "Alice", "role": "admin"},
    {"name": "Bob", "role": "user"},
    {"name": "Alice", "role": "admin"},
    {"name": "Charlie", "role": "user"}
]

def unique_dicts(items, key_func):
    seen = set()
    result = []
    for item in items:
        key = key_func(item)
        if key not in seen:
            seen.add(key)
            result.append(item)
    return result

unique_users = unique_dicts(users, lambda x: x["name"])
print(unique_users)
# Output: [{'name': 'Alice', 'role': 'admin'}, {'name': 'Bob', 'role': 'user'}, {'name': 'Charlie', 'role': 'user'}]

The key_func parameter lets you decide what counts as a duplicate. In this case, we're using the "name" field, but you could easily check against multiple fields or a computed value.

When You Need to Count Duplicates

Sometimes you don't just want to remove duplicates — you want to know how many times each item appears. The collections.Counter class is perfect for this:

from collections import Counter

items = [4, 2, 4, 1, 3, 2, 4, 1]
counts = Counter(items)
print(counts)  # Output: Counter({4: 3, 2: 2, 1: 2, 3: 1})

# Get unique items
unique_items = list(counts.keys())
print(unique_items)  # Output: [4, 2, 1, 3]

This gives you both the unique values and their frequencies. At PythonSkillset, we've used this pattern for everything from analyzing survey responses to cleaning up database exports.

The Manual Loop Approach

Sometimes you want full control over the deduplication process. Maybe you need to apply custom logic or handle edge cases. Here's a straightforward loop that's easy to modify:

def custom_deduplicate(items):
    unique = []
    for item in items:
        if item not in unique:
            unique.append(item)
    return unique

numbers = [1, 2, 2, 3, 4, 3, 5]
result = custom_deduplicate(numbers)
print(result)  # Output: [1, 2, 3, 4, 5]

This method is slower for large lists because checking item not in unique requires scanning the entire result list each time. But it's crystal clear in its intent and easy to customize. You could add logging, skip certain values, or apply transformations before checking for duplicates.

Using List Comprehension with enumerate()

For a more Pythonic approach that still preserves order, you can combine list comprehension with enumerate():

data = [10, 20, 10, 30, 20, 40]
unique = [data[i] for i in range(len(data)) if data[i] not in data[:i]]
print(unique)  # Output: [10, 20, 30, 40]

This checks each element against everything that came before it. It's elegant but not the most efficient for large lists since it does a linear search for each element. For lists under a few hundred items, it works just fine.

The itertools Approach for Large Datasets

When you're dealing with really large lists — think hundreds of thousands of items — you might want something more memory-efficient. The itertools module has a recipe for this:

from itertools import filterfalse

def unique_everseen(iterable, key=None):
    seen = set()
    seen_add = seen.add
    for element in filterfalse(lambda x: x in seen or seen_add(x), iterable):
        yield element

data = [1, 2, 2, 3, 4, 3, 5]
result = list(unique_everseen(data))
print(result)  # Output: [1, 2, 3, 4, 5]

This generator-based approach is memory efficient because it yields items one at a time. The filterfalse function skips elements that are already in the seen set, and the or trick ensures we add new items to the set while still returning them.

What About Nested Lists?

Removing duplicates from a list of lists requires a bit more thought because lists aren't hashable. Here's a solution that works:

nested = [[1, 2], [3, 4], [1, 2], [5, 6], [3, 4]]
unique_nested = list(set(tuple(sublist) for sublist in nested))
unique_nested = [list(item) for item in unique_nested]
print(unique_nested)  # Output: [[1, 2], [3, 4], [5, 6]]

We convert each inner list to a tuple (which is hashable), use a set to remove duplicates, then convert back to lists. This preserves the structure while eliminating duplicates.

Performance Considerations

Here's what I've learned from real-world use at PythonSkillset:

• For lists under 1000 items: Any method works fine. Use whatever is most readable.
• For lists between 1000 and 100,000 items: The set() approach or dict.fromkeys() are your best friends. They're both O(n) and very fast.
• For lists over 100,000 items: Stick with set() if order doesn't matter. If order matters, the dict.fromkeys() method is still efficient, but consider whether you really need to keep everything in memory.

A Word on Mutable Items

Remember that lists and dictionaries can't be used as set elements because they're mutable. If you're dealing with a list of lists, you'll need to convert them to tuples first:

list_of_lists = [[1, 2], [3, 4], [1, 2], [5, 6]]
unique = list(set(tuple(x) for x in list_of_lists))
unique = [list(x) for x in unique]
print(unique)  # Output: [[1, 2], [3, 4], [5, 6]]

This two-step conversion is a bit clunky but gets the job done. If you're working with complex nested structures, you might want to write a helper function that handles the conversion automatically.

Which Method Should You Choose?

Here's a quick guide based on what you need:

• Don't care about order, want speed: Use list(set(your_list))
• Need to preserve order: Use dict.fromkeys() or the loop with a set
• Working with unhashable items: Use the loop approach with a custom key function
• Need frequency counts: Use collections.Counter
• Teaching beginners or writing clear code: Use the manual loop — it's the most explicit

At PythonSkillset, we've found that the dict.fromkeys() method hits the sweet spot for most real-world scenarios. It's fast, preserves order, and reads naturally. But don't be afraid to use the simple set approach when order doesn't matter — it's hard to beat for raw speed.

One Last Tip

If you're working with data that might contain None values or other edge cases, test your chosen method first. Some approaches handle None differently, and you don't want surprises in production code. A quick unit test with your actual data types will save you headaches later.

Removing duplicates is one of those small tasks that makes your code cleaner and your data more reliable. Pick the method that fits your specific needs, and you'll be writing cleaner Python in no time.