Identify and Remove Duplicate Rows

You’ve spent hours cleaning a dataset—fixing types, renaming columns, filling missing values—and then you run a summary statistic and realize the numbers look… off. The same customer appears twice, the same transaction is logged three times, and your counts are inflated. Duplicate rows are silent saboteurs: they skew averages, overstate frequencies, and derail any analysis built on top of them. In this lesson, you’ll learn how to identify and remove duplicate rows in pandas quickly and accurately, so your downstream analysis is built on clean, trustworthy data.

The problem this lesson solves

Duplicate rows are more common than you think. They sneak in from:

• Merged datasets — joining multiple sources often creates repeated records
• Manual data entry — a user submits the same form twice
• Web scraping — refreshing a page can re-scrape identical rows
• Sensor logs — devices can send duplicate readings

Why does this matter? If you failed to remove duplicates, consider what happens:

• Aggregations lie: sum(), mean(), and count() treat each duplicate as a separate observation, inflating results
• Relationships break: joining on a key that has duplicates produces a Cartesian explosion
• Machine learning degrades: models learn from overrepresented rows, biasing predictions
• Reports mislead: stakeholders make decisions based on wrong numbers

In short, duplicates poison every step of the data analysis pipeline. The longer you wait, the more costly the bug becomes. By the end of this lesson, you’ll be able to identify them, remove them, and keep your analysis honest.

Core concept / mental model

Think of your DataFrame as a database table. Each row represents an observation (a customer, a sale, a reading). A duplicate is a row that repeats the same information—either exactly or for a subset of columns you care about.

Pandas gives you two core tools:

• duplicated() — a detective: flags all rows that are duplicates (returns a boolean Series)
• drop_duplicates() — a cleaner: removes those flagged rows (or keeps them, depending on how you set it up)

Here’s the mental image: imagine a guest list. If two people arrive with the exact same name and address, you have a duplicate. If you only care about names (maybe two different guests share a name), you can decide to treat one as a duplicate based on just that column. The default is to consider all columns—only rows that are identical across every field are considered duplicates.

Pro tip: The first occurrence is kept by default. The keep parameter lets you control which one to keep—first, last, or none.

How it works step by step

Let’s walk through the process logically.

Step 1: Inspect your data

Before removing anything, always look at what you’re dealing with. Use df.head() and df.shape to get a feel for the size and structure.

Step 2: Identify duplicates

Call df.duplicated() to get a boolean Series. Rows that are duplicates (occurring after the first occurrence) are marked True. You can quickly count them with .sum().

But not all duplicates are exact. Sometimes you only care about a subset of columns—like customer_id or order_id. Use the subset parameter to specify those columns.

Step 3: Decide on a removal strategy

• Keep the first (keep='first') — the most common choice; keeps the earliest record
• Keep the last (keep='last') — useful if later entries are more complete
• Drop all duplicates (keep=False) — if any duplicate exists, remove all copies; good for audit scenarios

Step 4: Remove and verify

Call drop_duplicates() (or drop_duplicates(subset=...)) to create a cleaned DataFrame. Then re-check with duplicated().sum() to confirm zero duplicates remain.

Step 5: Decide whether to mutate or copy

drop_duplicates() returns a new DataFrame by default. If you want to modify the original in place, set inplace=True—but be careful, in-place operations are often discouraged for clarity.

The key is to do no harm: copy, inspect, remove, and verify.

Hands-on walkthrough

Let’s put it all together with a real example.

Prepare the data

First, create a sample DataFrame with some duplicates.

import pandas as pd

# Sample data with duplicates
data = {
    'customer_id': [101, 102, 101, 103, 102, 104],
    'name': ['Alice', 'Bob', 'Alice', 'Charlie', 'Bob', 'Diana'],
    'purchase_amount': [250, 150, 250, 300, 150, 400]
}

df = pd.DataFrame(data)
print("Original shape:", df.shape)
print(df)

Expected output:

Original shape: (6, 3)
   customer_id    name  purchase_amount
0         101   Alice             250
1         102     Bob             150
2         101   Alice             250
3         103  Charlie             300
4         102     Bob             150
5         104   Diana             400

Rows 0 and 2 are identical, and rows 1 and 4 are identical.

Identify duplicates (all columns)

# Flag duplicate rows
duplicate_flags = df.duplicated()
print("Duplicate flags:")
print(duplicate_flags)
print("\nNumber of duplicates:", duplicate_flags.sum())

Expected output:

Duplicate flags:
0    False
1    False
2     True
3    False
4     True
5    False
dtype: bool

Number of duplicates: 2

Remove duplicates (exact matches)

# Drop exact duplicates, keeping the first occurrence
clean_df = df.drop_duplicates()
print("Cleaned shape:", clean_df.shape)
print(clean_df)

Expected output:

Cleaned shape: (4, 3)
   customer_id    name  purchase_amount
0         101   Alice             250
1         102     Bob             150
3         103  Charlie             300
5         104   Diana             400

Identify duplicates based on a subset of columns

Sometimes you only care about, say, customer_id. If a customer appears twice with different amounts, you might want to treat the second as a duplicate.

# Flag duplicates based on customer_id only
subset_flags = df.duplicated(subset=['customer_id'])
print("\nDuplicates based on customer_id:")
print(subset_flags)
print("Duplicates count:", subset_flags.sum())

Expected output:

Duplicates based on customer_id:
0    False
1    False
2     True
3    False
4     True
5    False
dtype: bool

Duplicates count: 2

Remove duplicates on a subset, keeping the last occurrence

clean_subset_last = df.drop_duplicates(subset=['customer_id'], keep='last')
print("\nAfter dropping (subset, keep last):")
print(clean_subset_last)

Expected output:

After dropping (subset, keep last):
   customer_id    name  purchase_amount
2         101   Alice             250
4         102     Bob             150
3         103  Charlie             300
5         104   Diana             400

Drop all duplicate rows (keep none)

# Drop every row that is part of a duplicate set
unique_only = df.drop_duplicates(keep=False)
print("\nRows without any duplicates:")
print(unique_only)

Expected output:

Rows without any duplicates:
   customer_id    name  purchase_amount
3         103  Charlie             300
5         104   Diana             400

Hands-on tip: After cleaning, always confirm with duplicated().sum() == 0.

Compare options / when to choose what

Different scenarios call for different keep and subset choices. Here’s a quick table:

Alternatives to pandas' built-in methods include:

• df[~df.duplicated()] — manual boolean filtering, useful in complex conditions
• groupby().first() or .last() — when you need a specific aggregate after deduplication
• SQL DISTINCT — if you move to a database, it’s the same concept

Troubleshooting & edge cases

Even with simple tools, things can go sideways. Here are common pitfalls and fixes.

Empty DataFrame after drop_duplicates(keep=False)

If you use keep=False on a dataset where every row appears at least twice, you’ll get an empty DataFrame. That’s expected but can be surprising. Check your data first:

print(df.duplicated().value_counts())

subset column contains NaN

Pandas treats NaN as a value; two rows with NaN in the subset column are considered duplicates. If you want to treat missing values as distinct, fill them before deduplication.

Fix: Use df[subset_col] = df[subset_col].fillna('') or a sentinel value.

Duplicates not removed because of invisible differences

A trailing space ('Alice ' vs 'Alice') or differing case can make two rows look unique. Normalize text first:

df['name'] = df['name'].str.strip().str.lower()
# then drop duplicates

inplace=True not working as expected

inplace=True modifies the original object, but it is easy to misuse. If you see no change, you may be calling it on a slice. Prefer returning a new object:

df_clean = df.drop_duplicates()  # safer, more readable

Duplicates keep reappearing in resampled data

If you join multiple tables and duplicates reappear, it’s often because you didn’t deduplicate the source tables before joining. Clean upstream first.

What you learned & what's next

You’ve mastered the essential skill of identifying and removing duplicate rows in pandas. You now know:

• How to use duplicated() to flag duplicates and count them
• How to use drop_duplicates() to remove them, with control over keep and subset
• How to choose between exact and subset-based deduplication
• How to troubleshoot common issues like text inconsistencies and NaN behavior

These are the building blocks of clean data—and clean data is the foundation of every reliable analysis. Next in the track, you’ll learn how to handle missing values with isna() and fillna(), another critical cleaning step. With duplicates gone, you’ll be ready to tackle incomplete data head-on.

Now that you can identify and remove duplicate rows, you’re one step closer to producing analysis that stakeholders can trust. Keep practicing—every dataset you clean makes you faster and more confident.