Summary Statistics in Python

You've cleaned your data, reshaped it into tidy tables, and maybe even spotted a few outliers by eye. But now you face the wall every data scientist hits: your boss wants a read on 50,000 rows right now, and you're about to write a for loop to compute averages column by column. Stop. Reaching for manual loops or spreadsheet formulas is the fastest way to burn an afternoon on something that should take you ten seconds. In this lesson, you'll learn how to create summary statistics quickly with pandas and NumPy — the one-liner toolkit that turns raw DataFrames into insight before your coffee cools.

The problem this lesson solves

Raw datasets are noisy, wide, and overwhelming. A DataFrame with 100 columns and 100,000 rows gives you no intuition — just a wall of numbers. To make decisions, you need a compressed view: one number per column that captures its center, spread, or distribution shape. That's a summary statistic.

The pain is real when you do it manually. A naive approach looks like this:

import pandas as pd

# Do NOT do this
df = pd.read_csv("sales.csv")
means = {}
for col in df.select_dtypes(include="number").columns:
    means[col] = df[col].sum() / df[col].count()
print(means)

That loop works, but it's slow to write, error-prone with missing values, and ignores the fact that pandas already has this logic baked in. The real problem: you're spending your analysis budget on plumbing instead of insight. Creating summary statistics quickly isn't a luxury — it's the difference between a 5-minute exploratory pass and a 2-hour slog.

Core concept / mental model

Think of a dataset as a landscape — a forest of values. A summary statistic is a compass reading that tells you something true about that forest without walking every tree.

• Central tendency tells you where the middle is: mean, median, mode.
• Spread tells you how tight the values cluster: standard deviation, variance, IQR.
• Shape hints at asymmetry or unusual peaks: skewness, kurtosis.
• Counts and missing values tell you how reliable your readings are.

Pandas gives you vectorized operations — a single .describe() or .agg() call that computes column-wise stats in C-optimized code, not Python loops. The mental model: one function call = one vectorized pass over the column, and the result is a new Series or DataFrame that you can chain into your next step.

Pro tip: If you can express a statistic as a function applied to a column, pandas can apply it to every column with .agg(). Vectorization isn't just fast — it's the cleanest way to write intention.

How it works step by step

Let's break down the workflow for creating summary statistics quickly from a fresh DataFrame.

Step 1: Load your data

Assume you have a CSV (or a DataFrame from any source). For this lesson, we'll generate a small demo dataset so you can run everything locally.

import pandas as pd
import numpy as np

# Create a sample DataFrame
df = pd.DataFrame({
    "price": [10.5, 12.0, 11.2, 14.1, 9.8, 11.5, 12.7],
    "quantity": [5, 7, 6, 8, 4, 6, 7],
    "category": ["A", "B", "A", "B", "A", "B", "A"]
})
print(df)

Step 2: Start with .describe()

The fastest starting point is df.describe() — it returns a DataFrame of count, mean, std, min, quartiles, and max for every numeric column.

print(df.describe())

Expected output (rounded):

Step 3: Customize with .agg()

.describe() is a great baseline, but you'll often need custom stats (median, mode, skewness). Use .agg() with a list of functions.

# Custom summary
def range_fn(s):
    return s.max() - s.min()

summary = df["price"].agg(["mean", "median", "std", range_fn, "skew"])
print(summary)

Step 4: Group-wise summaries

Summary statistics get really powerful when you split data by category.

grouped = df.groupby("category")["price"].agg(["mean", "median", "count"])
print(grouped)

Step 5: Visualize the summary (optional)

Once you have the stats, a quick matplotlib boxplot can confirm what the numbers tell you.

Hands-on walkthrough

Let's apply the workflow to a realistic scenario — an online store's daily sales. We'll load data, compute summary statistics quickly, and interpret them.

import pandas as pd
import numpy as np

# Simulate 100 days of sales data
np.random.seed(42)
dates = pd.date_range("2025-01-01", periods=100)
df = pd.DataFrame({
    "date": dates,
    "revenue": np.random.randint(1000, 5000, size=100),
    "orders": np.random.randint(50, 200, size=100)
})

print(df.head())

Step 1: Quick overview

print(df.describe())

Step 2: Add missing values to practice handling them (the real world is messy)

df.loc[5, "revenue"] = np.nan

Step 3: Compute stats ignoring NaN (pandas does this by default)

print(df["revenue"].mean())  # ignores NaN
print(df["revenue"].median())
print(df["revenue"].std())

Step 4: Custom summary for a column

summary = df["revenue"].agg(["mean", "median", "min", "max", "skew"])
print(summary)

Expected output (approximate):

mean      3103.343434
median    3050.500000
min       1002.000000
max       4992.000000
skew       -0.163824
Name: revenue, dtype: float64

Step 5: Group-wise summary (by weekday, for example)

df["weekday"] = df["date"].dt.day_name() grouped = df.groupby("weekday")["revenue"].agg(["mean", "count"]) print(grouped.sort_values("mean", ascending=False))

Then visualize:

```python
import matplotlib.pyplot as plt
df.boxplot(column="revenue", by="weekday")  # type: ignore
plt.show()

Compare options / when to choose what

You have several tools to create summary statistics quickly. Here's a comparison to help you pick the right one.

Rule of thumb:

• Need a quick answer? Use .describe().
• Need specific stats? Use .agg().
• Need per-group stats? Use groupby().agg().
• Working with raw NumPy arrays? Use np. functions but remember to drop NaNs with np.nanmean().

Pro tip: If you're in a Jupyter notebook, type df.describe().T to transpose and see columns as rows — easier for wide datasets.

Troubleshooting & edge cases

Even with one-liners, things can go wrong. Here are common pitfalls and fixes.

1. Missing values (NaN) are silently dropped by default

# pi = 3.14159, not NaN
print(df["revenue"].mean())

But np.mean() won't drop them — use np.nanmean().

import numpy as np
np.mean(df["revenue"])  # returns NaN
np.nanmean(df["revenue"])  # correct

2. Non-numeric columns cause errors

.describe() ignores non-numeric columns, but .agg() might fail if you apply a numerical function to text. Solution: filter with select_dtypes(include="number") first.

numeric_cols = df.select_dtypes(include="number").columns
df[numeric_cols].agg(["mean", "median"])

3. Groupby with NaN in the grouping column

Rows with NaN in the group column are dropped by default. If you need them included, pass dropna=False (pandas 1.1+).

df.groupby("category", dropna=False)["price"].mean()

4. My output has too many decimals

Use .round(2) on the result.

summary = df["price"].agg(["mean", "std"]).round(2)

5. Massive datasets — memory issues

If your DataFrame is huge, don't load everything; use pd.read_csv(..., usecols=[...]) and compute stats on chunks.

What you learned & what's next

You now know how to create summary statistics quickly — from a single .describe() to custom .agg() calls and grouped summaries. You can handle missing values, pick the right tool for the job, and avoid common pitfalls. That's the foundation for deeper exploration.

What you accomplished:

• Explained the core idea of summary statistics (central tendency, spread, shape)
• Completed a hands-on exercise with pandas and NumPy
• Connected the workflow to your data science pipeline

What's next: in the next lesson, you'll learn how to filter and transform columns efficiently — turning those summary insights into targeted slices of data for deeper analysis. But first, practice what you've learned!

Quick rehearsal: Take any CSV you have (or the sales demo above) and produce: 1) a full .describe(), 2) a custom .agg() that includes median and max, 3) a grouped summary by one categorical column. That's all it takes to become fluent in the 60-second data pulse.