dropna & fillna for Missing Values

You’ve finally got your hands on that shiny new dataset, only to realize it’s full of empty cells. Every time you try to calculate a mean, group by a category, or plot a trend, pandas throws NaN back at you, and your results start looking like Swiss cheese. This is the silent killer of data analysis — missing values that can skew your aggregates, break your visualizations, and lead you to completely wrong conclusions. In this lesson, you’ll master the two essential pandas methods to handle missing values with dropna and fillna, turning messy, incomplete data into a clean, analysis-ready DataFrame.

The problem this lesson solves

Missing data isn’t just an inconvenience — it’s a data quality issue that can corrupt your entire analysis. When cells in your DataFrame are NaN (Not a Number) or None, pandas treats them specially, but many operations either fail outright or silently produce misleading results.

Consider this common scenario: you’re analyzing customer orders and want to calculate the average order value. If a few order amounts are missing, a simple .mean() will skip those rows by default, giving you an answer that doesn’t reflect reality. Worse, if you try to plot a time series with gaps, your chart will have holes that obscure the true trend.

The core problem: you must decide what to do with missing values before you can trust any downstream analysis. Leaving them unchecked is not an option for robust work.

This lesson teaches you the two primary strategies to handle missing values with dropna and fillna:

• dropna — remove rows (or columns) that contain missing values.
• fillna — replace missing values with a specific value, a calculated statistic, or a forward/backward fill.

By the end, you’ll know when to drop, when to fill, and how to do both correctly in your data science workflow.

Core concept / mental model

Think of your DataFrame as a spreadsheet, and missing values as blank cells. You have two instincts when you see blanks:

1. Delete the row (or column) — the dropna approach.
2. Write something in the blank cell — the fillna approach.

Both are valid, but they answer different questions. dropna asks: “Can I still get reliable results if I remove these incomplete records?” fillna asks: “What’s the best guess I can make to preserve as much data as possible?”

Here’s a mental model to keep in mind:

• Dropping is like removing a bad apple from a basket — it’s fast, but you lose data.
• Filling is like repairing a cracked apple with wax — you keep the apple, but you’re adding something artificial.

The choice depends on your goals:

• If you have a small percentage of missing rows and removing them won’t bias your analysis, drop.
• If you have many missing values and dropping would lose too much information, fill — using a sensible replacement.

In pandas, missing values are typically represented by NaN (float) or None (object). Both are recognized by pandas.isna() and the dropna/fillna methods.

How it works step by step

Let’s dive into the mechanics. The dropna and fillna methods are part of pandas' Series and DataFrame objects. Here’s how they behave, step by step.

1. dropna() — Removing missing values

The simplest call — df.dropna() — removes any row that has at least one missing value. But you can fine‑tune it with parameters:

• axis — 0 for rows (default), 1 for columns.
• how — 'any' (default) drops if any value is missing; 'all' drops only if all values are missing.
• thresh — requires a minimum number of non‑missing values to keep the row/column.
• subset — list of column names to consider when looking for missing values.

Example: df.dropna(subset=['email']) removes rows where the email column is missing, even if other columns are complete.

2. fillna() — Replacing missing values

The fillna method replaces NaN with a value you specify. Key parameters:

• value — the fill value (scalar, dict, or Series).
• method — deprecated in pandas 2.x — instead use ffill or bfill.
• axis — direction for forward/backward fill.
• inplace — whether to modify the original DataFrame (deprecated; use assignment).

Common fills:

• Fill with a constant: df.fillna(0)
• Fill with a statistic: df['age'].fillna(df['age'].mean())
• Forward fill: df.fillna(method='ffill') — but in pandas 2.x, use df.ffill()
• Backward fill: df.bfill()

3. Understanding the return value

Both methods return a new DataFrame by default. The original remains unchanged unless you assign the result. This is a core part of the pandas API — always assign to a new variable or the same one to save the change.

# This does NOT modify df
new_df = df.dropna()

# This modifies df
new_df = df.dropna()

Hands-on walkthrough

Let’s put these methods into practice with a concrete example. We’ll create a small DataFrame with missing values and apply both dropna and fillna.

Setup and initial inspection

import pandas as pd
import numpy as np

# Create a DataFrame with missing values
data = {
    'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
    'age': [25, np.nan, 35, 29, np.nan],
    'salary': [50000, 60000, np.nan, 55000, 70000],
    'department': ['HR', 'IT', 'IT', np.nan, 'HR']
}

df = pd.DataFrame(data)
print(df)

Output:

      name   age   salary department
0    Alice  25.0  50000.0         HR
1      Bob   NaN  60000.0         IT
2  Charlie  35.0      NaN         IT
3    David  29.0  55000.0        NaN
4      Eve   NaN  70000.0         HR

Using dropna to remove incomplete rows

First, check how many missing values exist in each column:

print(df.isna().sum())

Output:

age           2
salary        1
department    1

Now remove rows with any missing values:

df_dropped_all = df.dropna()
print(df_dropped_all)

Output:

    name   age   salary department
0  Alice  25.0  50000.0         HR

Only Alice survives! That’s because every other row has at least one NaN. This is the aggressive default behavior.

What if you only care about age and salary?

df_dropped_subset = df.dropna(subset=['age', 'salary'])
print(df_dropped_subset)

Output:

      name   age   salary department
0    Alice  25.0  50000.0         HR
3    David  29.0  55000.0        NaN

Now David is kept because his missing department doesn’t matter for this filter.

Using fillna to replace missing values

Let’s fill the missing ages with the mean age:

mean_age = df['age'].mean()
df_filled_age = df.copy()
df_filled_age['age'] = df_filled_age['age'].fillna(mean_age)
print(df_filled_age)

Output:

      name   age   salary department
0    Alice  25.0  50000.0         HR
1      Bob  29.0  60000.0         IT
2  Charlie  35.0      NaN         IT
3    David  29.0  55000.0        NaN
4      Eve  29.0  70000.0         HR

Here, the missing ages (rows 1 and 4) are replaced by the mean age of 29.0.

For departments, you might want to fill with the mode (most frequent value):

mode_dept = df['department'].mode()[0]
df_filled_dept = df.copy()
df_filled_dept['department'] = df_filled_dept['department'].fillna(mode_dept)
print(df_filled_dept)

Output:

      name   age   salary department
0    Alice  25.0  50000.0         HR
1      Bob   NaN  60000.0         IT
2  Charlie  35.0      NaN         IT
3    David  29.0  55000.0         HR
4      Eve   NaN  70000.0         HR

Forward and backward fill for time series

If your data is ordered by time, a forward fill (carrying the last known value forward) is often a natural choice:

df_ffill = df.copy()
df_ffill = df_ffill.ffill()  # forward fill
print(df_ffill)

Output:

      name   age   salary department
0    Alice  25.0  50000.0         HR
1      Bob  25.0  60000.0         IT
2  Charlie  35.0  60000.0         IT
3    David  29.0  55000.0         IT
4      Eve  29.0  70000.0         HR

Notice how the missing age for Bob took Alice’s value (25), and missing salary for Charlie took Bob’s value (60000). This works well with sequential data.

Now try backward fill:

df_bfill = df.bfill()
print(df_bfill)

Output:

      name   age   salary department
0    Alice  25.0  50000.0         HR
1      Bob  35.0  60000.0         HR
2  Charlie  35.0  55000.0         IT
3    David  29.0  55000.0        NaN
4      Eve   NaN  70000.0         HR

Here, Bob’s missing age is filled with the next valid value (35 from Charlie), and Charlie’s missing salary is filled with David’s 55000. If the last row has a missing value, it stays NaN.

Compare options / when to choose what

Not every missing value needs the same treatment. Here’s a comparison to help you decide between dropna and fillna:

When to choose dropna:

• Missing values are few (say, <5% of rows).
• The missing rows are not systematically different from the rest.
• You need a clean, unbiased dataset for statistical tests.

When to choose fillna:

• Missing values are substantial, and dropping would lose critical data.
• You have a logical fill value (e.g., 0 for a count, median for income).
• You’re working with time series where continuity matters.

Variations and alternatives

• isna / notna — For filtering, you can use boolean masks: df[df['age'].notna()] is equivalent to df.dropna(subset=['age']) but gives you more control.
• interpolate() — For numeric data, you can linearly interpolate missing values: df['value'].interpolate().
• Scikit‑learn’s SimpleImputer — In a modeling pipeline, you might use SimpleImputer to fill missing values as part of preprocessing.

These alternatives give you a broader toolkit, but dropna and fillna remain the foundational methods.

Troubleshooting & edge cases

Even seasoned pandas users hit a few snags. Here are common mistakes and how to fix them:

Mistake: Using method parameter with pandas 2.x

# Deprecated in pandas 2.x → raises TypeError
df.fillna(method='ffill')

Fix: Use the direct method df.ffill() or df.bfill().

Mistake: Overwriting original data unexpectedly

df.fillna(0)  # no assignment — original unchanged
df.fillna(0, inplace=True)  # deprecated, but works in older versions

Fix: Always assign the result: df = df.fillna(0).

Mistake: Filling with the mean when data has outliers

If your column has large outliers, the mean might be misleading. Example: salaries [50000, 60000, NaN, 1000000]. The mean is ~370000 — a terrible fill.

Fix: Use the median for skewed data: df['salary'].fillna(df['salary'].median()).

Mistake: Dropping too much data

Calling dropna() on a wide DataFrame with many columns can delete 90% of your rows.

Fix: Use thresh to keep rows with at least a certain number of non‑missing values:

df.dropna(thresh=2)  # keep rows with at least 2 non-missing values

Edge case: All-missing columns

If a column is entirely NaN, df.dropna(axis=1) removes it, which is often desired. But beware — if you use thresh on columns, you might drop columns you actually need.

Edge case: None vs NaN

In object columns, None is treated as missing, but a string "None" is not. Always check with df.isna() to see what’s actually missing.

What you learned & what's next

You now know how to handle missing values with dropna and fillna in pandas. You’ve learned:

• The difference between dropping and filling missing data, and when to use each.
• How to use dropna with parameters like subset and thresh to control what gets removed.
• How to use fillna with constants, statistics, and forward/backward fills.
• Common pitfalls, like deprecated methods and data‑loss traps.

This skill is critical for any data analysis—you’ll use it in almost every real project. Next, you’ll learn how to handle categorical data or perform data transformations to further clean and prepare your data for analysis. Stay tuned!

Pro tip: Always inspect your data’s missing value pattern with df.isna().sum() before choosing a strategy. A quick count tells you whether dropping is safe or if you need to fill.