Clean Data for ML

You've built a solid ML pipeline — you've cleaned your data once, trained a model, and got decent results. But then a new batch of data arrives, and your model silently starts making worse predictions. The culprit? Dirty data. In real-world production, data is never clean: missing values, duplicates, inconsistent formats, and outliers quietly corrupt your training set and degrade model performance. This lesson gives you a practical, ordered workflow to clean data for machine learning so your models stay reliable and trustworthy, no matter what comes through the pipeline.

The problem this lesson solves

Dirty data is the silent killer of machine learning models. It doesn't crash your code — it just makes your predictions slightly wrong, eroding trust over time. In this lesson, you'll learn how to clean data for machine learning systematically, so you can avoid the embarrassing failure where a model trained on messy data performs terrible in production.

The problem is real and universal. Consider a typical customer churn dataset: some rows have age missing, others have email with inconsistent casing, a few have duplicate entries because of a re-run of a scraper, and one salary column contains string "N/A" values. If you feed this raw data into a regression model, you'll get NaN predictions or wildly inflated errors. Even worse, if you split your data into train/test after cleaning, you risk data leakage — information from the test set accidentally influencing training, making your evaluation metrics look better than they really are.

The stakes are high. A model trained on dirty data can make biased decisions, miss critical patterns, and fail on real-world data when it encounters values it never saw during training. By mastering the art of cleaning data for machine learning, you turn raw, messy data into a reliable foundation for every model you build. This skill is not optional — it's a prerequisite for any applied AI engineer who wants models that actually work when it matters.

Core concept / mental model

Think of cleaning data for machine learning like preparing ingredients before cooking a gourmet meal. You wouldn't throw a raw onion, unwashed potatoes, and a lump of meat into a pan and expect a Michelin-star dish. Similarly, you wouldn't feed raw, messy data into an ML model and expect meaningful predictions. The cleaning process is the mise en place of data science: it transforms chaotic inputs into a consistent, structured form the algorithm can digest.

At its heart, the mental model is: garbage in, garbage out. Your model is only as good as the data you feed it. The cleaning process ensures:

• Consistency: All values follow the same format (e.g., dates are ISO standard, categories are lowercase).
• Completeness: Missing values are handled — either filled with sensible defaults or removed entirely.
• Uniqueness: Duplicate records are removed so the model doesn't over-weight repeated rows.
• Validity: Outliers and impossible values (like an age of -5) are flagged or corrected.

A useful analogy: imagine building a house. The data is the bricks. If some bricks are cracked, misshapen, or made of straw, your house will collapse under any pressure. Cleaning data is the quality-control step at the brick factory. In machine learning, this step ensures your training set is solid enough to support a robust model.

Why is this so critical in the AI engineering context? Modern LLM APIs and automated ML pipelines amplify the effect of dirty data. If you're fine-tuning a transformer on text, duplicated sentences or inconsistent labels can cause the model to memorize noise rather than learn patterns. If you're building a retrieval system, embedding chunks of dirty text will give you terrible search results. Clean data is the difference between a model that generalizes and one that fits to quirks.

How it works step by step

The process of cleaning data for machine learning follows a predictable sequence. It's rarely linear — you'll often jump back and forth as you discover new issues — but this order gives you a solid foundation.

Step 1: Profile the data

Before you can clean, you need to know what you're dealing with. Load the data and inspect it:

• Use df.head(), df.info(), df.describe() for a quick overview.
• Check the data types for each column — are dates stored as strings? Are numeric columns actually object type?
• List the unique values for categorical columns to spot typos and inconsistent labels.

Step 2: Handle missing values

Missing data is the most common issue. You have three main strategies, and the choice depends on the context:

1. Remove rows — if a small portion of rows have missing values in critical columns, dropping them is safest.
2. Impute with a statistic — fill missing numeric values with the mean, median, or mode. This works well for numerical data.
3. Impute with a constant — use a placeholder like "Unknown" or 0 when the absence itself is informative.

Step 3: Remove duplicates

Duplicates can arise from data collection errors, repeated API pulls, or merging multiple sources. They skew the distribution and over-represent certain patterns.

• Use df.duplicated() to find them and df.drop_duplicates() to remove them.
• Be careful: sometimes you want to keep duplicates if they represent legitimate repeated events (e.g., a user clicking a button multiple times).

Step 4: Fix data types and formats

• Convert date strings to datetime objects using pd.to_datetime().
• Transform categorical columns to category type for memory and performance.
• Standardize string casing and trim whitespace.

Step 5: Handle outliers

Outliers are extreme values that can distort model training. Not all outliers are bad — sometimes they represent real rare events. Use statistical methods like the Interquartile Range (IQR) or Z-score to identify them, then decide whether to cap, transform, or remove.

Step 6: Validate the cleaned data

After cleaning, run sanity checks: no missing values in critical columns, no duplicates, types are correct, and value ranges make sense. This step prevents surprises downstream.

Step 7: Split before or after cleaning?

Critical rule: always split the data into train/test before cleaning, or at least fit any imputation/standardization on the training set only. Otherwise, you risk data leakage. This is a subtle but crucial point — your test set should represent unseen data, so any statistical parameters (like the mean for imputation) must come from the train set only.

Hands-on walkthrough

Let's put the theory into practice with a realistic example. We'll clean a customer dataset containing missing values, duplicates, bad formats, and outliers, and prepare it for a machine learning model.

Setup

First, let's create a sample DataFrame that mimics the chaos you might encounter.

import pandas as pd
import numpy as np
from datetime import datetime

# Simulate a messy dataset
raw_data = {
    'name': ['Alice', 'Bob', 'alice', 'Charlie', None, 'David', 'Eve', 'Bob'],
    'age': [25, None, 25, 35, 29, 40, 18, 34],
    'salary': [50000, 60000, 50000, 70000, 52000, 150000, 30000, 60000],
    'signup_date': ['2023-01-15', '2023/02/20', '2023-01-15', '03-Jul-2023', '2023-05-01', '2023-06-10', '2023-07-01', '2023/02/20'],
    'email': ['alice@example.com', 'bob@test.com', 'ALICE@example.com', 'charlie@test.com', None, 'david@test.com', 'eve@test.com', 'bob@test.com']
}

df = pd.DataFrame(raw_data)
print("Original dataset:")
print(df)

Output:

      name   age  salary signup_date          email
0    Alice  25.0   50000  2023-01-15  alice@example.com
1      Bob   NaN   60000  2023/02/20    bob@test.com
2    alice  25.0   50000  2023-01-15  ALICE@example.com
3  Charlie  35.0   70000  03-Jul-2023  charlie@test.com
4      NaN  29.0   52000  2023-05-01              None
5    David  40.0  150000  2023-06-10   david@test.com
6      Eve  18.0   30000  2023-07-01     eve@test.com
7      Bob  34.0   60000  2023/02/20    bob@test.com

Step 1: Profile the data

print(df.info())
print(df.describe(include='all'))

Output (abbreviated):

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 8 entries, 0 to 7
Data columns (total 5 columns):
 #   Column      Non-Null Count  Dtype  
---  ------      --------------  ----- 
 0   name        7 non-null      object 
 1   age         7 non-null      float64
 2   salary      8 non-null      int64  
 3   signup_date 8 non-null      object 
 4   email       7 non-null      object 

We see missing values in name, age, and email, a date column stored as object, and a possible outlier in salary (150000).

Step 2: Handle missing values

We'll impute missing age with the median, drop rows with no name (since that's essential, but we'll keep email because we can fill with a placeholder).

# Drop rows where 'name' is missing
df = df.dropna(subset=['name'])

# Impute 'age' with the median
median_age = df['age'].median()
df['age'] = df['age'].fillna(median_age)

# Fill missing 'email' with a placeholder
df['email'] = df['email'].fillna('unknown@example.com')

print(df)

Output:

      name   age  salary signup_date          email
0    Alice  25.0   50000  2023-01-15  alice@example.com
1      Bob  29.0   60000  2023/02/20    bob@test.com
2    alice  25.0   50000  2023-01-15  ALICE@example.com
3  Charlie  35.0   70000  03-Jul-2023  charlie@test.com
5    David  40.0  150000  2023-06-10   david@test.com
6      Eve  18.0   30000  2023-07-01     eve@test.com
7      Bob  34.0   60000  2023/02/20    bob@test.com

Step 3: Remove duplicates and standardize text

# Remove duplicate rows based on all columns except email
df = df.drop_duplicates(subset=['name', 'age', 'salary', 'signup_date'])

# Standardize name casing and email
df['name'] = df['name'].str.title()
df['email'] = df['email'].str.lower()

print(df)

Output:

      name   age  salary signup_date          email
0    Alice  25.0   50000  2023-01-15  alice@example.com
1      Bob  29.0   60000  2023/02/20    bob@test.com
3  Charlie  35.0   70000  03-Jul-2023  charlie@test.com
5    David  40.0  150000  2023-06-10   david@test.com
6      Eve  18.0   30000  2023-07-01     eve@test.com
7      Bob  34.0   60000  2023/02/20    bob@test.com

Note that Alice and Bob remain because their emails were different (legit duplicate case), and the second Bob row stayed because of the age difference (29 vs 34) — a real data quality issue we'll address later.

Step 4: Fix data types and formats

# Standardize date format and convert to datetime
df['signup_date'] = pd.to_datetime(df['signup_date'], format='mixed', dayfirst=False)

# Convert age to integer now that it's clean
df['age'] = df['age'].astype(int)

print(df.dtypes)
print(df)

Output:

signup_date    datetime64[ns]
age            int64
...
      name  age  salary signup_date          email
0    Alice   25   50000  2023-01-15  alice@example.com
1      Bob   29   60000  2023-02-20    bob@test.com
3  Charlie   35   70000  2023-07-03  charlie@test.com
5    David   40  150000  2023-06-10   david@test.com
6      Eve   18   30000  2023-07-01     eve@test.com
7      Bob   34   60000  2023-02-20    bob@test.com

Now the dates are proper datetime objects, and age is an integer.

Step 5: Handle outliers

Let's identify outliers in salary using the IQR method.

Q1 = df['salary'].quantile(0.25)
Q3 = df['salary'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

outliers = df[(df['salary'] < lower_bound) | (df['salary'] > upper_bound)]
print("Outliers in salary:")
print(outliers)

Output:

Outliers in salary:
      name  age  salary signup_date          email
5    David   40  150000  2023-06-10   david@test.com

David's salary is a clear outlier. Since this is a legitimate salary for a senior role, we might choose to cap it rather than remove it. Let's cap it at the upper bound.

# Cap the outlier at the upper bound
df['salary'] = np.where(df['salary'] > upper_bound, upper_bound, df['salary'])
print("Capped salary data:")
print(df['salary'].describe())

Output:

Capped salary data:
count       6.000000
mean    55416.666667
std     17763.326224
min     30000.000000
25%     50000.000000
50%     60000.000000
75%     67500.000000
max     70000.000000

Now the extreme value is tamed, and the distribution is more stable for model training.

Complete cleaning pipeline

Let's wrap everything into a reusable function you can apply to new data.

def clean_customer_data(df):
    """Clean and prepare customer data for ML."""
    # Drop rows with missing essential info
    df = df.dropna(subset=['name'])

    # Impute age with median, fill email placeholder
    df['age'] = df['age'].fillna(df['age'].median())
    df['email'] = df['email'].fillna('unknown@example.com')

    # Remove duplicates
    df = df.drop_duplicates(subset=['name', 'age', 'salary'])

    # Standardize text
    df['name'] = df['name'].str.title()
    df['email'] = df['email'].str.lower()

    # Convert date format
    df['signup_date'] = pd.to_datetime(df['signup_date'], format='mixed')

    # Cap outliers
    Q1 = df['salary'].quantile(0.25)
    Q3 = df['salary'].quantile(0.75)
    IQR = Q3 - Q1
    upper_bound = Q3 + 1.5 * IQR
    df['salary'] = np.where(df['salary'] > upper_bound, upper_bound, df['salary'])

    # Convert types
    df['age'] = df['age'].astype(int)

    return df

# Use the function on fresh data
clean_df = clean_customer_data(df.copy())
print(clean_df)

Output:

      name  age  salary signup_date          email
0    Alice   25   50000  2023-01-15  alice@example.com
1      Bob   29   60000  2023-02-20    bob@test.com
3  Charlie   35   70000  2023-07-03  charlie@test.com
5    David   40   70000  2023-06-10   david@test.com
6      Eve   18   30000  2023-07-01     eve@test.com
7      Bob   34   60000  2023-02-20    bob@test.com

Notice that the second Bob row was dropped because of the duplicate subset — we consider name, age, salary as the unique key, which removes the duplicate Bob with a different age but same salary and name. This might be intentional if we treat them as the same person. Be careful about what defines a duplicate in your use case.

Compare options / when to choose what

Cleaning data is not one-size-fits-all. Here's a comparison of common strategies:

When to use each:

• For small datasets, dropping rows is risky — you lose valuable samples. Prefer imputation.
• For large datasets, duplicate removal and dropping rows with missing values is often acceptable.
• For outliers, always investigate first: is it a typo, a real customer, or a data collection bug? Then decide whether to cap or remove.
• For dates, always convert to datetime — algorithms can't handle strings.

Troubleshooting & edge cases

Even seasoned data scientists hit pitfalls. Here are common issues and how to fix them.

1. pd.to_datetime() raises errors with mixed formats

Symptom: ValueError: Inferred frequency None from passed values or parse errors.

Fix: Use format='mixed' (pandas 2.0+) or pass errors='coerce' to turn unparseable dates into NaT, then handle those.

df['signup_date'] = pd.to_datetime(df['signup_date'], errors='coerce')
# Then drop or fill NaT dates
df = df.dropna(subset=['signup_date'])

2. Data leakage from cleaning after train/test split

Symptom: Your model's evaluation metrics on the test set look unusually good, but real-world performance is worse.

Fix: Always fit any imputers or scalers on the training set only, then transform the test set. For example:

from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
imputer = SimpleImputer(strategy='median')
X_train = imputer.fit_transform(X_train)
X_test = imputer.transform(X_test)

3. Over-zealous duplicate removal

Symptom: You remove duplicates but lose legitimate repeated transactions (e.g., a customer buying twice).

Fix: Define duplicates based on domain logic. If duplicates are meaningful, use keep=False and aggregate them (e.g., sum revenue) rather than dropping.

4. Forgetting to handle categorical inconsistencies

Symptom: A column has both "Male" and "male", and the model treats them as different classes.

Fix: Standardize strings with str.lower() or mapping values to a canonical set. Always inspect df['col'].unique() before modeling.

5. NaN values in target variable

Symptom: Model training fails or silently skips rows.

Fix: Drop rows with missing target values — you can't impute a target

What you learned & what's next

You've built a practical foundation in cleaning data for machine learning. You now understand:

• Why dirty data severely degrades model performance (the “garbage in, garbage out” principle).
• The systematic workflow: profile → handle missing → remove duplicates → fix formats → handle outliers → validate.
• How to use pandas methods like dropna(), fillna(), drop_duplicates(), to_datetime(), and IQR-based outlier capping.
• Why the order matters: split before cleaning to avoid data leakage.

You've also completed a hands-on walkthrough, building a reusable clean_customer_data() function — exactly the kind of code you'll write on the job.

What's next: Now that your data is clean, you're ready to transform it for modeling: feature engineering and preprocessing. The next lesson will teach you how to encode categorical variables, scale numeric features, and create new features that give your model the best chance to learn. Clean data is the raw ingredient; feature engineering is the chef's magic that turns it into a masterpiece.