Explore Data with Pandas

You’ve cleaned your data, maybe even loaded it into a DataFrame, but now comes the moment of truth: what’s actually in there? Raw numbers and columns don’t tell a story until you explore them. Without a systematic approach to data exploration, you’ll miss outliers, misunderstand distributions, and build models on shaky foundations. This lesson gives you a repeatable workflow to explore data with pandas and visualizations — so you can spot trends, catch errors early, and make confident decisions before writing a single model.

The problem this lesson solves

Every data science project starts with a question, but the data rarely answers it directly. The file you load might have missing values, skewed distributions, or variables that don’t mean what their names suggest. If you jump straight to modeling, you risk:

• Silent data corruption — a -999 placeholder that looks like a number but represents “missing.”
• Wrong assumptions about distributions — algorithms like linear regression expect roughly Gaussian features, but you won’t know yours aren’t without plotting.
• Feature-target relationships hiding in plain sight — a scatter plot can reveal a correlation that a correlation matrix misses.

Exploratory Data Analysis (EDA) is the process of summarizing, visualizing, and sanity-checking your data before modeling. It’s the difference between guessing and knowing. In this lesson, you’ll build a repeatable EDA workflow using pandas for tabular inspection and Matplotlib/Seaborn for visual patterns.

Core concept / mental model

Think of pandas as your data microscope — it lets you zoom into rows, columns, and summary statistics with surgical precision. Visualizations are your data photos — they show you the big picture that numbers alone hide.

The workflow follows a simple loop:

1. Load the data into a DataFrame.
2. Inspect structure (shape, dtypes, head/tail).
3. Summarize with descriptive statistics.
4. Visualize distributions and relationships.
5. Clean based on what you find.

💡 Pro tip: Always start with df.head() and df.info(). They tell you if your data loaded correctly and whether columns have the right types — a step that saves hours of debugging later.

A solid mental model: exploration is a detective process. Each check (summary, plot, value count) is a clue. Your job is to follow the clues until the data’s story becomes clear.

How it works step by step

Let’s break down the core operations you’ll use in every EDA session.

1. Inspect the structure

• df.shape — number of rows and columns
• df.columns — column names
• df.dtypes — data types (int, float, object, datetime)
• df.head(n) / df.tail(n) — peek at first/last rows

These commands tell you if your data loaded correctly and if column names match your expectations.

2. Summarize with descriptive statistics

df.describe() gives count, mean, std, min, quartiles, and max for numeric columns. This is your first sanity check:

• Are the min/max values reasonable?
• Is the mean close to the median (or heavily skewed)?
• Are there NaN counts you didn’t expect?

3. Check for missing values

df.isnull().sum() tells you exactly which columns have gaps. Missing data is a silent killer — you must decide whether to fill, drop, or flag it before modeling.

4. Analyze categorical variables

df['column'].value_counts() shows frequency distributions. This reveals imbalances (e.g., 99% one category) that could bias your model.

5. Visualize

Visualization turns numbers into insight. We’ll cover histograms, box plots, scatter plots, and correlation heatmaps in the next section.

Hands-on walkthrough

Now let’s apply this to a real dataset. We’ll use the classic Iris dataset — you can load it directly with seaborn.

First, make sure you have the libraries installed:

pip install pandas matplotlib seaborn

Load and inspect

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Load the Iris dataset
df = sns.load_dataset('iris')

# Inspect structure
print(df.shape)
print(df.dtypes)
print(df.head())

Expected output:

(150, 5)
sepal_length    float64
sepal_width     float64
petal_length    float64
petal_width     float64
species          object

dtypes: float64(4), object(1)
memory usage: 1.2+ KB
   sepal_length  sepal_width  petal_length  petal_width species
0           5.1          3.5           1.4          0.2  setosa
1           4.9          3.0           1.4          0.2  setosa
2           4.7          3.2           1.3          0.2  setosa
3           4.6          3.1           1.5          0.2  setosa
4           5.0          3.6           1.4          0.2  setosa

The data loaded successfully — 150 rows, 5 columns, and the types look correct.

Summarize and check missing values

# Descriptive stats
print(df.describe())

# Missing values
print(df.isnull().sum())

Expected output (abridged):

       sepal_length  sepal_width  petal_length  petal_width
count    150.000000   150.000000    150.000000  150.000000
mean       5.843333     3.057333      3.758000    1.199333
std        0.828066     0.435866      1.765298    0.762238
min        4.300000     2.000000      1.000000    0.100000
25%        5.100000     2.800000      1.600000    0.300000
50%        5.800000     3.000000      4.350000    1.300000
75%        6.400000     3.300000      5.100000    1.800000
max        7.900000     4.400000      6.900000    2.500000

sepal_length    0
sepal_width     0
petal_length    0
petal_width     0
species         0
dtype: int64

No missing values — great. Now let’s visualize.

Create a distribution plot

# Histogram for sepal length
plt.figure(figsize=(8, 5))
sns.histplot(df['sepal_length'], bins=20, kde=True)
plt.title('Distribution of Sepal Length')
plt.xlabel('Sepal Length (cm)')
plt.show()

Expected output: a bell-shaped curve centered around 5.8 cm — roughly normal, but with a slight right skew.

Create a box plot to detect outliers

plt.figure(figsize=(8, 5))
sns.boxplot(x='species', y='sepal_length', data=df)
plt.title('Sepal Length by Species')
plt.show()

Expected output: three box plots. Setosa is clearly shorter, Versicolor in the middle, Virginica tallest — a strong signal that species predicts sepal_length.

Visualize relationships with a scatter plot

plt.figure(figsize=(8, 6))
sns.scatterplot(x='sepal_length', y='petal_length', hue='species', data=df)
plt.title('Sepal Length vs Petal Length')
plt.show()

Expected output: points cluster by species — setosa in the bottom-left, versicolor and virginica forming an upward trend. This suggests a near-linear relationship between these two features.

Compare options / when to choose what

You have multiple visualization libraries and techniques. Here’s a quick guide:

Pro tip: For large datasets (>100k rows), always downsample or use kernel density plots. Full scatter plots become unreadable ink blobs.

Troubleshooting & edge cases

Even with clean data, things go wrong. Here’s how to fix common issues.

Error: TypeError: 'numpy.ndarray' object is not callable

This happens when you accidentally shadow df.plot with a variable named plot. Fix: restart your kernel or rename the variable.

Wrong: Histogram shows one giant bar

Your data might have a few extreme outliers that squash the rest. Fix: use a log scale — plt.xscale('log') — or clip the data to the 1st–99th percentile.

Wrong: Scatter plot shows no pattern

Sometimes patterns are hidden by overplotting (too many points). Fix: add alpha=0.3 to make points translucent, or use sns.kdeplot for a density view.

Error: ImportError: No module named 'seaborn'

You forgot to install it. Run pip install seaborn in your terminal or notebook cell.

Missing values cause plots to break?

matplotlib can’t handle NaN in some functions. Drop them first: df = df.dropna() — but only if you’ve confirmed the percentage is small.

What you learned & what's next

You now have a repeatable EDA workflow: load, inspect, summarize, visualize, and interpret. You can identify distributions, spot outliers, and see relationships before modeling. That’s the foundation of every applied AI project.

Next lesson: We’ll take these insights and start building features for a machine learning model — using what you discovered to create meaningful predictors. You’ll learn how to encode categorical variables, handle missing data, and split your dataset for training and testing.

Keep practicing on your own data — the more you explore, the better your models will be.