Missing Values: fillna & dropna

You've finally cleaned your dataset, merged a few CSV files, and then you see it: NaN scattered like confetti across your DataFrame. Worse, your regression model just threw a cryptic error, or your group-by summary silently dropped half your rows. Missing values aren't just an inconvenience — they're a silent killer of data integrity. In this lesson, you'll master pandas' two most essential tools for dealing with the void: fillna() for replacing missing values and dropna() for removing them. You'll learn not just how to use them, but when to use each, based on your data and the question you're trying to answer.

The problem this lesson solves

Real-world data is messy. Sensors fail, survey respondents skip questions, and joins between tables produce gaps. If you ignore this, you'll face a cascade of issues:

• Broken computations: Many pandas functions, like mean() or sum(), return NaN if any input is missing, or silently exclude those rows, skewing your results.
• Errors in Machine Learning: Most scikit-learn estimators refuse to work with NaN values, throwing ValueError: Input contains NaN.
• Skewed visualizations: Plotting a column with gaps leaves holes in your lines or bars, hiding true trends.

Before pandas, data scientists spent hours writing custom loops to check and handle missing data. The fillna() and dropna() methods abstract away that pain, giving you a clean, expressive, and vectorized way to deal with missingness. Understanding them is non-negotiable for any data scientist working with pandas.

Core concept / mental model

Think of your DataFrame as a spreadsheet where some cells are empty. dropna() is the equivalent of deleting entire rows (or columns) that contain at least one empty cell. fillna() is like filling those empty cells with a post-it note: a default value, the average of the column, or the value from the previous row.

A simple mental model:

• dropna() — Out of sight, out of mind. Use when you can afford to lose data and you're confident the missing values are random and not informative.
• fillna() — Patch it up. Use when you need to keep the row's other data, and you have a sensible substitute for the missing value.

Both methods return a new DataFrame by default, leaving your original data untouched. This is a core pandas principle — you almost never mutate data in place unless you explicitly use inplace=True (which is often discouraged). The how, thresh, and subset parameters give you fine-grained control, and method lets you forward-fill or backward-fill, which is crucial for time-series data.

How it works step by step

Both methods work on the entire DataFrame, or a specific column. Here's the typical flow:

1. Detect missing values — Use df.isnull() or df.isna() (they're aliases) to create a boolean mask. Summarize with df.isnull().sum() to see the count per column.

2. Decide on a strategy — Determine whether to drop rows, drop columns, or fill values. This depends on (a) the percentage of missing data, (b) the importance of the column, and (c) whether the missingness is random.

3. Apply dropna() or fillna() — Choose the right parameters. For dropna(), decide if you want to drop any row with at least one NaN (how='any') or only rows where all values are missing (how='all'). For fillna(), choose a scalar, a method, or a column-specific dictionary.

4. Verify the result — Check df.isnull().sum() again to ensure the data is clean as expected.

Let's see this in action.

Hands-on walkthrough

First, create a sample DataFrame with missing values. In a real scenario, you'd load it from a CSV, but this mirrors the structure.

import pandas as pd
import numpy as np

# Sample dataset with missing values
df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
    'age': [25, np.nan, 30, 29, np.nan],
    'salary': [50000, 60000, np.nan, 55000, 52000],
    'city': ['NY', 'LA', 'NY', np.nan, 'LA']
})

print(df)
print("\nMissing values per column:\n", df.isnull().sum())

Expected output:

      name   age   salary  city
0    Alice  25.0  50000.0    NY
1      Bob   NaN  60000.0    LA
2  Charlie  30.0      NaN    NY
3    David  29.0  55000.0   NaN
4      Eve   NaN  52000.0    LA

Missing values per column:
 name      0
age       2
salary    1
city      1

Dropping rows with dropna()

The simplest use: drop every row with any missing value.

# Drop rows where any value is missing (default)
df_drop_any = df.dropna()
print(df_drop_any)

Output:

    name   age   salary city
0  Alice  25.0  50000.0   NY

Only Alice's row survives, because all others contain at least one NaN. That's aggressive — you lost 80% of your data! Usually you'd only do this when missingness is rare.

You can be more targeted with the subset parameter, focusing on specific columns that are critical.

# Drop rows only if 'age' is missing
df_drop_age = df.dropna(subset=['age'])
print(df_drop_age)

Output:

      name   age   salary  city
0    Alice  25.0  50000.0    NY
2  Charlie  30.0      NaN    NY
3    David  29.0  55000.0   NaN

Now only Bob and Eve (who lack age) are removed, preserving records that might still be useful despite other gaps.

Filling missing values with fillna()

Now let's patch instead of delete. Use a scalar to fill all missing values with 0, or fill each column with its mean.

# Fill all missing values with 0
df_fill_zero = df.fillna(0)
print(df_fill_zero)

# Fill numeric columns with their column mean
df_fill_mean = df.copy()
df_fill_mean['age'] = df_fill_mean['age'].fillna(df_fill_mean['age'].mean())
df_fill_mean['salary'] = df_fill_mean['salary'].fillna(df_fill_mean['salary'].mean())
print(df_fill_mean)

Output (for the mean version):

      name        age   salary  city
0    Alice  25.000000  50000.0    NY
1      Bob  28.000000  60000.0    LA
2  Charlie  30.000000  53500.0    NY
3    David  29.000000  55000.0   NaN
4      Eve  28.000000  52000.0    LA

Note that the mean of age is (25 + 30 + 29) / 3 = 28.0, and the mean of salary is (50000 + 60000 + 55000 + 52000) / 4 = 53500.0. Using the column mean is a common imputation trick, but it reduces variance — be aware of that.

For categorical columns, a more honest approach is to fill with a placeholder like 'Unknown' or the mode (most frequent value).

# Fill categorical with mode and numeric columns via dictionary
df_fill_mode = df.copy()
df_fill_mode['city'] = df_fill_mode['city'].fillna(df_fill_mode['city'].mode()[0])
df_fill_mode['age'] = df_fill_mode['age'].fillna(df_fill_mode['age'].median())
print(df_fill_mode)

The missing city becomes 'LA' (the mode), and missing ages become the median (28.0 as well, same result here). Notice how fillna accepts a dictionary that maps column names to values — that's your scalpel for per-column strategies.

Forward-fill for time series

The method parameter is a lifesaver for time-series data (e.g., stock prices, sensor logs). ffill propagates the last valid observation forward, and bfill does the opposite.

# Time series with a missing 'temperature'
time_df = pd.DataFrame({
    'time': pd.date_range('2024-01-01', periods=5, freq='h'),
    'temp': [71, np.nan, 73, 74, np.nan]
})

# Forward-fill
ffilled = time_df['temp'].fillna(method='ffill')
print(ffilled)

Output:

0    71.0
1    71.0
2    73.0
3    74.0
4    74.0

The missing values inherit the last known temperature, which mimics real-world sensor behavior. Pro tip: method='ffill' is equivalent to method='pad', and it's often used with limit=1 to avoid filling too many gaps in a row.

Blockquote: Pro tip Always inspect df.isnull().sum() before and after handling missing values. A quick sanity check prevents silent data loss and ensures your imputation worked as expected.

Compare options / when to choose what

Key decision guide: If missing values are truly random and you have plenty of data, dropna() is safe. If data is scarce, or you want to keep every observation, use fillna(). For time-series, forward-fill is your best friend — but don't fill more than a few time steps in a row.

Troubleshooting & edge cases

• Error: ValueError: Fill value must be a scalar, dict or DataFrame — You tried fillna with a list. Wrap it in a dictionary: df.fillna({'col': [1,2,3]}) is invalid; use a scalar or per-column scalars.
• Warning: The 'method' keyword in fillna is deprecated (use .ffill()) — In recent pandas versions, method='ffill' is being phased out. Use the explicit .ffill() and .bfill() methods instead.
• Data type changes after fillna(0) — Filling integers with 0 usually preserves dtype, but filling with a float (like the mean) converts the column to float. That's expected — be ready for it.
• dropna() silently deletes rows — After a drop, your DataFrame index is renumbered (unless you set ignore_index=True). Old row numbers are gone, so don't refer to them later.
• Filling a column with its own mean using inplace=True — You might accidentally mutate the original during imputation, making it hard to trace steps. Prefer returning copies.

What you learned & what's next

You've now got two battle-tested methods in your pandas toolkit. You can inspect missing data with isnull(), remove rows or columns with dropna(), and impute gaps with fillna() using scalars, statistics, or forward-fill. You also know when to drop vs. fill, and the operational pitfalls to avoid. This is a standalone skill, but it plugs directly into your next lesson where you'll learn data transformation and feature engineering — turning raw columns into model-ready inputs. With clean data, you can confidently move on to reshaping, encoding categories, and scaling numeric features, all of which assume your data is free of missing values.