Explore Data with Pandas Profiling

You’ve got a brand-new dataset—maybe a CSV you scraped or a table from a database—and the first thing you need to do is explore data with pandas profiling. The pain is real: raw df.describe() only gives you basic stats, and writing your own loop to check dtypes, missing values, and outliers eats up hours. Pandas-profiling generates an interactive HTML report with every insight you need—distributions, correlations, missing values, and warnings—in a single line of code. In this lesson, you’ll learn how to use it, what it tells you, and when to switch to a lighter alternative.

The problem this lesson solves

Exploratory Data Analysis (EDA) is the first step in any Applied AI engineering project. Without a thorough EDA, you risk feeding garbage into your model. But traditional EDA is slow and repetitive: you call df.head(), df.info(), df.describe(), then manually plot histograms for every column. For a dataset with 50 columns, that’s hours of boilerplate work. Pandas profiling replaces that effort with an automated, one-shot report that answers most of your initial questions before you write a single custom script.

Why it matters now: In real production pipelines, data arriving from APIs, logs, or data lakes is often messy—missing values, inconsistent types, and outliers. You need a fast way to sanity-check a dataset before you invest in feature engineering or model training. This is exactly where pandas-profiling shines.

Core concept / mental model

Think of pandas-profiling as a data detective that automatically interviews every column in your DataFrame. It computes a suite of statistics, detects patterns, and compiles everything into a clear HTML report.

Key components of the report: - Overview: dataset statistics, number of variables, missing values, and duplicate rows. - Variables: per-column details—type, unique values, missing percentage, mean, min/max, and a histogram. - Correlations: heatmaps for Pearson, Spearman, and other methods to spot relationships. - Missing values: a matrix view showing where data is absent. - Warnings: a flag for high cardinality, skewed distributions, or missing data—perfect for quick data-quality checks.

Analogy: Imagine you’re a doctor receiving a patient. Instead of checking each vital sign one by one, pandas-profiling hands you a full health chart—blood pressure, heart rate, bloodwork—all in one readout. You still need to interpret it, but you don’t waste time measuring each metric manually.

How it works step by step

1. Load your data into a pandas DataFrame (from CSV, database, or API).
2. Generate a profile using ProfileReport(df).
3. Render the report as HTML or as a Jupyter notebook object.
4. Inspect the report to identify data quality issues, distributions, and correlations.
5. Act on findings—clean data, transform columns, or select features for modeling.

The library computes everything in one pass, using pandas under the hood. It’s not magic—it’s a set of statistical summaries and visualizations packaged into a reusable component.

Hands-on walkthrough

Let’s get your hands dirty. First, install the library. I recommend version 3.6.6 (or newer 3.6.6+ versions) for Python 3.10+ compatibility:

pip install pandas-profiling

Note: pandas-profiling was recently renamed to ydata-profiling (as of v4.x). For new projects, install ydata-profiling and import pandas_profiling for backward compatibility—we’ll cover that in variations.

Now generate a profile for a sample dataset. We’ll use the built-in Titanic dataset (available via seaborn) to mimic a real-world problem:

import pandas as pd
import seaborn as sns
from pandas_profiling import ProfileReport

# Load a classic dataset
df = sns.load_dataset('titanic')
print(df.head())

# Generate the profile
profile = ProfileReport(df, title="Titanic Data Profiling Report")

# Save to an HTML file
profile.to_file("titanic_report.html")

Expected output: When you open titanic_report.html, you’ll see: - 891 rows, 15 columns - age has 177 missing values (19.9% missing) - deck has 688 missing values (77.2% missing)—a clear warning - Correlation heatmap shows fare and pclass are strongly negatively correlated - Histograms for every numeric column

You can also render the report inline in a Jupyter notebook:

# In a Jupyter cell
profile

To see a quick text summary (for CI or debugging), you can convert the profile to a dictionary:

summary = profile.get_description()
print(summary['missing'])
# Output: {'age': 177, 'embark_town': 2, 'deck': 688, 'embarked': 2}

Now, let’s customize the report to focus on specific columns or types. For example, you might want to ignore certain columns that are irrelevant (like passwords or IDs):

profile = ProfileReport(df, title="Titanic Report",
                        vars={"num": {"low_categorical_threshold": 0}, "cat": {"n_obs": 5}})
profile.to_file("custom_report.html")

The vars parameter lets you control how numeric and categorical variables are treated—perfect for large or mixed datasets.

Hands-on tip: Always save the report to an HTML file so you can share it with your team or revisit it later without rerunning the script.

Compare options / when to choose what

Pandas-profiling is powerful, but it’s not the only option. Here’s a comparison with common alternatives:

When to use pandas-profiling: - You’re starting a new project and need a broad overview. - You’re training a model and want to detect cleaning tasks (missing values, outliers). - You need a shareable report for stakeholders.

When to skip it: - You have millions of rows—sampling may be needed. - You only need a quick numeric check—stick with df.describe().

Variations: ydata-profiling and df.describe()

Variation 1: ydata-profiling As of 2023, the original pandas-profiling was renamed to ydata-profiling. The usage is nearly identical:

pip install ydata-profiling
from ydata_profiling import ProfileReport
profile = ProfileReport(df, title="Report")

Variation 2: Minimal alternative If you don’t want a full report, df.describe() plus df.isna().sum() gives a quick text summary:

print(df.describe())
print(df.isna().sum())

This is fine for a quick check, but you lose the visual context.

Troubleshooting & edge cases

Error: ImportError: cannot import name 'ProfileReport' from 'pandas_profiling'

This usually happens when you installed ydata-profiling but imported the old name without the compatibility layer. Either install pandas-profiling (v3.6.6) or use the new import:

# Fix 1: pip install pandas-profiling==3.6.6
# Fix 2: from ydata_profiling import ProfileReport

Problem: Report generation is slow on a large dataset

Pandas-profiling computes correlations and histograms for every column—O(nm) operations. If you have 1M rows and 50 columns, it may take minutes. Fix:* take a random sample:

sample = df.sample(n=50000, random_state=42)
profile = ProfileReport(sample, title="Sampled Report")

Issue: Missing values are not shown for object columns

If a column contains empty strings ("") instead of NaN, pandas-profiling won’t flag them as missing. Fix: preprocess to convert empty strings to NaN:

df.replace("", pd.NA, inplace=True)

Edge case: Duplicate rows inflate warnings

Profiling counts duplicates, but sometimes duplicates are valid (e.g., repeated events). Check the overview section and decide if you need to drop_duplicates().

What you learned & what's next

You’ve learned to explore data with pandas profiling — a tool that automates EDA and gives you a comprehensive, shareable report. You can now: - Generate a one-line profiling report for any DataFrame. - Interpret the key sections: overview, variables, correlations, missing values, and warnings. - Customize the report to ignore variables or adjust thresholds. - Troubleshoot common import and performance issues.

This skill is a prerequisite for the next lesson in the Applied AI engineering path: data cleaning and feature engineering. You’ll take the insights you gained from profiling (missing values, outliers, correlations) and turn them into actionable transformations—like imputing missing ages or encoding categorical variables—to prepare data for model training. You’ll also learn how to integrate profiling into an automated pipeline to catch data drift.

Now go ahead—load a dataset you care about, generate a report, and let it guide your next cleaning step.