Missing Values: dropna & fillna

You’ve spent hours cleaning a dataset, and just when you think it’s ready, you run a calculation and get a wall of NaN or a cryptic error. Missing values are the silent saboteurs of data analysis — they skew averages, break machine learning models, and turn a simple .sum() into a confusing mess. In this lesson, you’ll learn how to handle missing values with dropna and fillna, the two pandas workhorses that let you remove or replace NaN values with confidence. By the end, you’ll know exactly when to drop, when to fill, and how to avoid the common traps that trip up even experienced analysts.

The problem this lesson solves

Real-world data is messy. Surveys have skipped questions, sensors go offline, and databases store NULL values. When you load that data into pandas, missing entries become NaN (Not a Number). If you ignore them, you’ll get misleading results — for example, df['salary'].mean() silently ignores missing values, but df.describe() might show a count different from your expectations. Worse, many scikit-learn models refuse to train on data with any missing values, throwing an error like Input contains NaN. This lesson gives you the tools to handle missing values with dropna and fillna, so you can clean your data, avoid surprises, and keep your analysis moving forward.

Missing values don’t just annoy you — they actively corrupt your work. A single NaN can turn a sum into NaN if you’re not careful, and a pivot table can produce empty cells that break your visualizations. Instead of panicking when you see NaN, you’ll learn to inspect, decide, and act. This is the problem this lesson solves: turning missing data from a blocker into a decision point.

Core concept / mental model

Think of your DataFrame as a table with some cells left blank. dropna is the eraser — it removes rows (or columns) that contain missing values. fillna is the ink pot — it fills those blanks with a value you choose, like a number, a string, or a computed statistic.

Here’s a simple mental model:

• dropna: “I don’t want incomplete records — remove them.”
• fillna: “I want to keep my records — let me fill the gaps with something sensible.”

Both methods are part of the pandas library and work on DataFrame and Series objects. The key is knowing when to use each. If missing values are rare and random, dropping might be fine. If missing values are common or systematic, filling is often better to preserve your sample size.

A helpful analogy: imagine you’re taking a class survey. If a few students didn’t answer “favorite color,” you might drop those rows because they’re incomplete. But if half the class skipped “salary,” you’d be throwing away valuable data — better to fill the missing salaries with the median to keep your dataset usable.

In pandas, missing values are typically represented as NaN (float) or None. dropna and fillna treat them the same way, so you don’t have to worry about the type — just know that NaN is the standard for missing numeric data.

How it works step by step

Let’s break down how to handle missing values with dropna and fillna, step by step.

Step 1: Identify missing values

Before you drop or fill, you need to know where the gaps are. Use df.isna() to get a boolean mask, or df.isna().sum() to count missing values per column. This tells you the scope of the problem.

Step 2: Decide whether to drop or fill

Ask yourself:

• Is the missing data random or systematic?
• How many rows/columns are affected?
• What’s the impact on your analysis?

If only a few rows have missing values, dropping might be fine. If many rows are affected, filling is safer.

Step 3: Use dropna() to remove missing data

dropna() by default removes any row that contains at least one NaN. But you can customize:

• axis=1 to drop columns instead of rows.
• how='all' to only drop rows where all values are missing.
• thresh=n to keep rows with at least n non-missing values.

Step 4: Use fillna() to replace missing values

fillna() lets you fill with a constant, a method (like forward or backward fill), or a statistic like the mean or median. For example:

• df['col'].fillna(0) fills with 0.
• df.fillna(method='ffill') forward-fills the last valid value.
• df['col'].fillna(df['col'].mean()) fills with the column mean.

Important: By default, both dropna and fillna return a new object. If you want to modify your DataFrame in place, use inplace=True (though modern pandas encourages reassigning instead).

Step 5: Verify the result

After dropping or filling, confirm there are no missing values left using df.isna().sum() again. This is a simple sanity check that your cleaning worked.

Hands-on walkthrough

Let’s put this into practice with a real example. We’ll create a small DataFrame with missing values and clean it using dropna and fillna.

Example 1: Inspect and drop missing rows

import pandas as pd
import numpy as np

# Create a sample DataFrame with missing values
df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'David'],
    'age': [25, np.nan, 30, 35],
    'salary': [50000, 60000, np.nan, 65000]
})

print("Original DataFrame:")
print(df)
print("\nMissing values per column:")
print(df.isna().sum())

# Drop rows with any missing value
df_cleaned = df.dropna()
print("\nAfter dropna (default - any missing):")
print(df_cleaned)

Expected output:

Original DataFrame:
      name   age   salary
0    Alice  25.0  50000.0
1      Bob   NaN  60000.0
2  Charlie  30.0      NaN
3    David  35.0  65000.0

Missing values per column:
name      0
age       1
salary    1
dtype: int64

After dropna (default - any missing):
    name   age   salary
0  Alice  25.0  50000.0
3  David  35.0  65000.0

Example 2: Fill missing values with a constant and with the mean

import pandas as pd
import numpy as np

# Reuse the same DataFrame
df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'David'],
    'age': [25, np.nan, 30, 35],
    'salary': [50000, 60000, np.nan, 65000]
})

# Fill missing age with 0, missing salary with the mean
df_filled = df.copy()
df_filled['age'] = df_filled['age'].fillna(0)
df_filled['salary'] = df_filled['salary'].fillna(df_filled['salary'].mean())

print("After fillna with constant and mean:")
print(df_filled)

Expected output:

After fillna with constant and mean:
      name   age   salary
0    Alice  25.0  50000.0
1      Bob   0.0  60000.0
2  Charlie  30.0  58333.333333
3    David  35.0  65000.0

Example 3: Forward fill for time series

import pandas as pd
import numpy as np

# Simulate a time series with missing values
dates = pd.date_range('2024-01-01', periods=5, freq='D')
df_ts = pd.DataFrame({
    'date': dates,
    'value': [100, np.nan, 102, np.nan, 104]
})

# Forward fill missing values
df_ts['value'] = df_ts['value'].fillna(method='ffill')
print("After forward fill:")
print(df_ts)

Expected output:

After forward fill:
        date  value
0 2024-01-01  100.0
1 2024-01-02  100.0
2 2024-01-03  102.0
3 2024-01-04  102.0
4 2024-01-05  104.0

Pro tip: Always check the result after filling with statistical values like the mean. The mean is sensitive to outliers, so in skewed data, the median is often a better choice.

Compare options / when to choose what

You now have two main tools. Here’s a comparison to help you decide when to use each.

Variations to consider

• Use limit: In fillna, you can limit how many consecutive values to fill with limit=2, which prevents carrying values too far.
• Drop columns instead of rows: If a column is mostly missing, you might prefer dropna(axis=1, how='all') to remove it entirely.
• Interpolation: For numeric data, df.interpolate() estimates missing values based on surrounding points — a slick alternative to simple filling.

Troubleshooting & edge cases

Even with the right tools, you’ll hit snags. Here are common issues and how to fix them.

1. dropna() removes too many rows

Symptom: Your DataFrame shrinks dramatically after dropna(). Fix: Only drop rows where certain critical columns are missing. Use subset=['column_name'] to limit the check:

df.dropna(subset=['salary'])

2. fillna() doesn’t fill because the values are not NaN

Sometimes missing values appear as empty strings '' or 'N/A'. fillna() won’t catch them. Convert them first:

df.replace('N/A', np.nan, inplace=True)

3. method='ffill' doesn’t fill the first rows

If the first row has a missing value, forward fill can’t handle it. Use bfill instead, or fill with a default first:

df['col'].fillna(method='bfill')

4. inplace=True gives unexpected behavior

Modifying in place can cause surprises if you reuse the DataFrame. Prefer reassigning:

df = df.fillna(0)

5. dropna() drops columns unexpectedly

When you set axis=1, it removes columns with any missing value. Be explicit with how to avoid surprises:

df.dropna(axis=1, how='all')

What you learned & what's next

You now know how to handle missing values with dropna and fillna. You learned to:

• Explain the core idea: dropna removes incomplete rows/columns, while fillna fills gaps with constants, statistics, or forward/backward values.
• Complete a practical exercise: you inspected missing values, dropped rows, filled missing ages with 0, salaries with the mean, and applied forward fill for time series.
• Connect this skill to the next lesson in the track: once your data is clean and complete, you’re ready to perform more advanced transformations, like grouping, merging, or building visualizations with Matplotlib and Seaborn.

Missing value handling is a cornerstone of data cleaning — master it, and your analyses will be far more reliable. Next, we’ll explore how to combine and reshape datasets, building on the clean, complete data you now know how to create.