Handle Missing Values with fillna and dropna

You've spent hours cleaning data, only to discover that a handful of NaN values just threw off your entire analysis. Maybe your averages are skewed, your plots have gaps, or your machine learning model refuses to run. Missing data is one of the most common — and most annoying — issues in real-world datasets. But with pandas, you have two powerful tools to handle missing values: fillna() and dropna(). By the end of this lesson, you'll know exactly when to fill gaps with sensible values and when to drop rows altogether, turning messy data into a clean, analysis-ready DataFrame.

The problem this lesson solves

Real-world data is rarely perfect. Surveys skip questions, sensors fail, and log files omit fields. In pandas, missing data is represented as NaN (Not a Number) or None. If you ignore them, you'll get:

• Wrong summary statistics — mean() and sum() silently skip NaN, giving you a biased result.
• Broken visualizations — gaps or jagged lines in your charts.
• Errors in downstream code — many functions, especially in scikit-learn, simply refuse to operate on data that contains NaN.

So the problem is: how do you make missing values disappear in a way that doesn't distort your analysis or crash your pipeline? This lesson gives you the two core strategies: fill the gap with a substitute value, or drop the incomplete row or column.

Core concept / mental model

Think of your dataset as a spreadsheet with a few empty cells. You have two choices: you can write something in the blank cell (filling it) or cut out the entire row that contains the blank (dropping it). Both are valid, but they answer different questions:

• Fill — "I believe the missing value is close to what I already have, so I'll estimate it."
• Drop — "I don't trust this row enough, so I'll remove it entirely."

Pandas gives you two methods that map directly to these choices:

• DataFrame.fillna(value, method, ...) — replaces every NaN with a specified value, a computed statistic like the mean, or a forward/backward fill.
• DataFrame.dropna(axis, how, thresh, ...) — removes rows (or columns) that contain missing values.

The key is knowing when to use which. If you've got 1000 rows and only 5 missing values, dropping is trivial. But if 30% of your rows are missing a column, dropping would throw away too much valuable data — you need to fill.

How it works step by step

1. Detect missing values first

Before you fix anything, you need to find out where the gaps are. Use isna() or its alias isnull() to get a boolean mask, and then summarize with sum() to see the count per column.

2. Decide on a strategy

Ask yourself: Is the missing data random, or is there a pattern? If it's random and sparse, dropping might be fine. If it's systematic (e.g., older entries lack a new column), filling is better.

3. Apply dropna() or fillna()

For dropna():

• Set axis=0 (default) to drop rows, axis=1 to drop columns.
• Use how='any' (default) to drop if any value is missing, how='all' to drop only if all values are missing.
• Use thresh=n to keep rows with at least n non-missing values.

For fillna():

• Pass a constant: df.fillna(0).
• Pass a per-column dict: df.fillna({'age': 30, 'income': 0}).
• Use a statistic: df.fillna(df.mean()).
• Use forward fill: df.fillna(method='ffill') to carry the last known value forward, or method='bfill' to use the next value.
• Use inplace=True to modify the DataFrame directly (though chaining is safer).

4. Verify your result

After filling or dropping, run df.isna().sum().sum() to ensure no missing values remain. If you're filling, check that your new values don't distort your data (e.g., mean before vs after).

Hands-on walkthrough

Let's start with a small DataFrame that simulates a common scenario: sales data with missing values in different columns.

import pandas as pd
import numpy as np

# Create a DataFrame with missing values
df = pd.DataFrame({
    'product': ['A', 'B', 'C', 'D', 'E'],
    'price': [100, np.nan, 150, 200, np.nan],
    'quantity': [5, 10, np.nan, 8, 12],
    'region': ['North', 'South', np.nan, 'East', 'West']
})

print(df)
print("\nMissing count per column:")
print(df.isna().sum())

Output:

  product  price  quantity region
0       A  100.0       5.0  North
1       B    NaN      10.0  South
2       C  150.0       NaN    NaN
3       D  200.0       8.0  East
4       E    NaN      12.0  West

Missing count per column:
price      2
quantity   1
region     1
dtype: int64

Now, let's apply different strategies:

Option 1: Drop rows with any missing value

# Drop any row that has at least one NaN
df_drop = df.dropna()
print(df_drop)

Output:

  product  price  quantity region
0       A  100.0       5.0  North
3       D  200.0       8.0  East

We lost 3 out of 5 rows — not great if this were a real dataset.

Option 2: Fill with column means

# Fill numeric columns with their mean, and categorical with a placeholder
df_filled = df.copy()
df_filled['price'].fillna(df['price'].mean(), inplace=True)
df_filled['quantity'].fillna(df['quantity'].mean(), inplace=True)
df_filled['region'].fillna('Unknown', inplace=True)

print(df_filled)

Output:

  product       price  quantity  region
0       A  100.000000       5.0  North
1       B  150.000000      10.0  South
2       C  150.000000       8.75  Unknown
3       D  200.000000       8.0  East
4       E  150.000000      12.0  West

Now we kept all rows, but the filled values are estimates (mean price is 150, mean quantity is 8.75).

Option 3: Forward fill for time-series-like data

# Add a date column and sort by it

df_t = pd.DataFrame({
    'date': pd.date_range('2025-01-01', periods=5, freq='D'),
    'temperature': [22, np.nan, 24, np.nan, 26]
})

df_t['temperature_ffill'] = df_t['temperature'].fillna(method='ffill')
print(df_t)

Output:

        date  temperature  temperature_ffill
0 2025-01-01         22.0               22.0
1 2025-01-02          NaN               22.0
2 2025-01-03         24.0               24.0
3 2025-01-04          NaN               24.0
4 2025-01-05         26.0               26.0

Pro tip: Use method='ffill' for time series where you want to carry the last known value forward. For the first row with a missing value, it stays NaN — you may need a separate fillna() after.

Compare options / when to choose what

Variations to consider:

• fillna(method='bfill') — for when the next value is a better predictor (e.g., in reverse-ordered data).
• df.interpolate() — linear interpolation for time series, a middle ground between ffill and a mean.
• Use limit in fillna(method='ffill', limit=2) to cap how many consecutive NaN get filled, preventing over-propagation.

Troubleshooting & edge cases

1. inplace not working as expected

• Problem: df.fillna(0, inplace=True) seems to do nothing.
• Cause: The column might be of object dtype, or you're using a chain (like df[df['col']>0].fillna()), which works on a copy.
• Fix: Use df = df.fillna(0) instead of inplace to avoid chained assignment issues.

2. Mean fill on integer columns turns them into float

• Cause: Mean of integers is a float; pandas promotes the column to float64.
• Fix: Round after filling: df['col'] = df['col'].fillna(df['col'].mean()).round(), or use median and cast back to int.

3. Forward fill leaves leading NaNs

• Cause: ffill has no previous value to use.
• Fix: Chain fillna(method='ffill').fillna(method='bfill') or fill with a constant first.

4. Dropping based on a specific column only

• Problem: You want to drop rows only if a certain column is NaN.
• Solution: Use df.dropna(subset=['price']) — this only checks that column.

# Drop rows where 'price' is NaN, regardless of other columns
df_filtered = df.dropna(subset=['price'])
print(df_filtered)

Output:

  product  price  quantity region
0       A  100.0       5.0  North
2       C  150.0       NaN    NaN
3       D  200.0       8.0  East

What you learned & what's next

You now know how to handle missing values with fillna and dropna — the two pillars of data cleaning in pandas. You can detect missing data with isna(), choose a strategy based on how much data is missing, and apply fillna() with constants, statistics, or forward/backward fills, or use dropna() to remove incomplete rows and columns. You also know how to handle common pitfalls like leading NaNs and dtype changes.

You're now ready to tackle more advanced cleaning scenarios. In the next lesson, you'll learn how to handle duplicates — another essential step to ensure your data is truly analysis-ready. You'll use drop_duplicates() to identify and remove duplicate rows, and you'll know how to combine it with your missing-value skills to clean even the messiest datasets.

Next up: Detecting and removing duplicate records to finish your data cleaning toolkit.