Handle Missing Values with Imputation Strategies

You've got your dataset loaded, your head is full of ideas, and then it hits you: NaN, None, or those ugly blank cells staring back from your DataFrame. It's the silent killer of data analysis — one missing value can skew your averages, break your visualizations, or cause your model to fail entirely. The question isn't if you'll encounter missing data; it's when. This lesson shows you how to handle missing values with imputation strategies that are practical, flexible, and grounded in real-world data analysis. You'll learn not just how to fill gaps but why each strategy makes sense for different situations, and you'll walk away with code you can use immediately.

The problem this lesson solves

Missing data is everywhere. A survey respondent skips a question, a sensor fails for a minute, a legacy system stored 0 for values that simply weren't recorded. Left unchecked, these gaps can silently distort your analysis. For example:

• Statistical measures like the mean or median become biased if you drop rows without thinking.
• Machine learning models often refuse to work with NaN values, forcing you to handle them before training.
• Visualizations can show misleading trends when missing points are simply left out.

Dropping data (removing rows with any missing value) is the default reflex for many beginners, but it's a blunt instrument. If you drop 30% of your rows because they have one missing cell, you lose valuable information and introduce bias. Imputation—replacing missing values with estimated ones—lets you keep your data intact and your analyses reliable.

But imputation isn't a one-size-fits-all solution. If you blindly fill every NaN with the column mean, you can create a false sense of precision and even destroy the natural variance of your data. The real skill is knowing which strategy to apply, and that's exactly what this lesson gives you.

Core concept / mental model

Think of your dataset as a restaurant menu. Some dishes are fully described, but a few have "price on request" or "ask server" where the cost should be. You have two choices: remove those dishes from the menu (dropping) or make a reasonable guess based on similar dishes (imputation). Imputation is like asking the chef what the price would be, based on ingredients and portion size.

In pandas, missing values appear as NaN (for float columns), None (for object columns), or sometimes as empty strings or sentinel values like -999. The first step is always to detect them with isnull() or isna().

Imputation strategies fall into three broad families:

• Simple imputation — replace with a single statistic (mean, median, mode, or a constant).
• Contextual imputation — use values from neighboring rows (like forward-fill for time series).
• Model-based imputation — predict the missing values using other features (e.g., linear regression, KNN).

For most data analysis tasks, simple and contextual methods are enough. Model-based approaches are powerful but add complexity and risk overfitting if not done carefully.

How it works step by step

Here's a mental workflow you can apply to any dataset:

1. Detect missing values: use df.isnull().sum() to see how many gaps exist per column.
2. Understand the nature: is the missingness random? Does it follow a pattern? For example, a column might be missing for all rows after a certain date — that’s a clue.
3. Choose a strategy based on data type and context: - Numeric columns: mean or median (median is safer for skewed data). - Categorical columns: mode (most frequent) or a new "missing" category. - Time series: forward-fill or interpolation.
4. Apply the transformation to your DataFrame using pandas or scikit-learn.
5. Verify the result: check that missing values are gone and that summary statistics haven't changed drastically.

This sequence ensures you don't just fill gaps blindly — you make a deliberate, defensible choice.

Hands-on walkthrough

Let's get practical. We'll use a small dataset to demonstrate the most common imputation strategies. First, install pandas if you haven't already (pip install pandas), then run the following in a Jupyter notebook or script.

Step 1: Load and inspect

import pandas as pd
import numpy as np

# Create a sample DataFrame with missing values
np.random.seed(42)
data = {
    'age': [25, 30, np.nan, 35, 40, np.nan, 22],
    'income': [50000, 60000, 55000, np.nan, 70000, 65000, 45000],
    'city': ['NYC', 'LA', 'SF', 'NYC', np.nan, 'LA', 'SF']
}
df = pd.DataFrame(data)

print(df)

Expected output:

   age   income  city
0 25.0  50000.0  NYC
1 30.0  60000.0  LA
2  NaN  55000.0  SF
3 35.0      NaN  NYC
4 40.0  70000.0  NaN
5  NaN  65000.0  LA
6 22.0  45000.0  SF

Now check the missing-value count:

print(df.isnull().sum())

Expected output:

age      2
income   1
city     1
dtype: int64

Step 2: Simple imputation with mean, median, and mode

For the age column, the mean is __— let's calculate and fill:

# Mean imputation for numeric columns
mean_age = df['age'].mean()
df['age_filled_mean'] = df['age'].fillna(mean_age)

# Median imputation (more robust to outliers)
median_income = df['income'].median()
df['income_filled_median'] = df['income'].fillna(median_income)

# Mode imputation for categorical data
mode_city = df['city'].mode()[0]  # mode() returns a Series, take first value
df['city_filled_mode'] = df['city'].fillna(mode_city)

print(df[['age_filled_mean', 'income_filled_median', 'city_filled_mode']])

Expected output (values will match your random seed):

   age_filled_mean  income_filled_median city_filled_mode
0             25.0                55000.0              NYC
1             30.0                60000.0              LA
2             30.0                55000.0              SF
3             35.0                60000.0              NYC
4             40.0                70000.0              LA
5             30.0                65000.0              LA
6             22.0                45000.0              SF

Notice how the median filled the missing income with 60000 (a robust choice since income data often has outliers).

Step 3: Forward-fill for time series

If your data is sequential (e.g., daily stock prices), a missing value often means "same as last known value." Pandas makes this easy:

# Simulate a time series
ts = pd.Series([100, 105, np.nan, 110, np.nan, 120])
ts_ffill = ts.ffill()

print(ts_ffill)

Expected output:

0    100.0
1    105.0
2    105.0
3    110.0
4    110.0
5    120.0
dtype: float64

For more complex interpolation (linear or polynomial), use interpolate():

print(ts.interpolate())

Expected output:

0    100.0
1    105.0
2    107.5
3    110.0
4    115.0
5    120.0
dtype: float64

Step 4: Using scikit-learn for consistent pipelines

For model-ready preprocessing, use SimpleImputer — it can fit on training data and transform test data, which is crucial to avoid data leakage.

from sklearn.impute import SimpleImputer
import numpy as np

# For numeric columns
imputer = SimpleImputer(strategy='median')
X = df[['age', 'income']].values
X_imputed = imputer.fit_transform(X)

# For categorical columns
cat_imputer = SimpleImputer(strategy='most_frequent')
cities = df[['city']].values
cities_imputed = cat_imputer.fit_transform(cities)

print(X_imputed)
print(cities_imputed)

Expected output:

[[25. 50000.]
 [30. 60000.]
 [30. 55000.]
 [35. 60000.]
 [40. 70000.]
 [30. 65000.]
 [22. 45000.]]
[['NYC']
 ['LA']
 ['SF']
 ['NYC']
 ['LA']
 ['LA']
 ['SF']]

Pro tip: Always fit your imputer on the training set only, then apply it to the test set. This prevents your model from 'seeing' test data during training, which would give you false confidence in your accuracy.

Compare options / when to choose what

When to choose what? If you're doing exploratory analysis, median/mean is fine. For modeling, consider model-based imputation, but simpler often wins. For time series, forward-fill is your best friend. There's no universal answer — always test how imputation affects your downstream results.

Troubleshooting & edge cases

• All values in a column are missing — If a column has no non-null values, mean() returns NaN, and mode() returns an empty Series. Check df.isnull().all() before imputing.
• Constant filling misfires — Using fillna(0) for a column like income can be disastrous; the zeros look like real values and skew your stats. Always compare before/after.
• Categorical mode gives multiple values — mode() can return multiple modes. Use [0] to pick the first, but be aware that this is arbitrary.
• Data leakage in pipelines — Validating a model with imputation fitted on the test set inflates accuracy. Use cross-validation and Pipeline to avoid this.
• Mixed types — If a column has both numbers and text, pandas will treat it as object dtype. Coerce with pd.to_numeric(errors='coerce') after cleaning.
• Time series gaps — Forward-fill misleads if there are long gaps; consider using limit parameter to avoid filling too far.

What you learned & what's next

You now understand how crucial it is to handle missing values, and you've mastered the core imputation strategies: mean, median, mode, forward-fill, and even scikit-learn's SimpleImputer. You can detect gaps with isnull(), choose a method based on data type and context, and verify your results. This is a foundational skill that elevates every subsequent analysis you do.

In the next lesson, you'll build on this by exploring feature scaling and normalization — transforming your data so models perform better. With missing values handled, your data will be ready for the next step in the pipeline.

Keep practicing: take any dataset, introduce missing values, and apply different imputation strategies. Compare how each affects summary statistics and simple visualizations. The more you experiment, the more instinctive these choices become.