Group Data with Groupby

You have a DataFrame full of rows — sales, users, sensor readings — and you need answers like "What's the average revenue per region?" or "How many users signed up each day?" Doing this with loops and filters is slow, error-prone, and scales terribly when your data grows to millions of rows. The groupby operation in pandas is the missing key that unlocks split-apply-combine: split your data into groups, apply a function to each group, and combine the results. In this lesson, you'll master groupby operations — from basic grouping to advanced aggregations — so you can summarize and slice your data with confidence.

The problem this lesson solves

Raw tabular data rarely gives you the insight you need. Instead, you need summaries by category: daily sales totals, average rating per product, median income per city. Without groupby, you'd write fragile loop-based code that is verbose and slow:

# Painful manual grouping
import pandas as pd

df = pd.DataFrame({'region': ['North', 'South', 'North', 'East'],
                   'sales': [100, 200, 150, 300]})

regions = df['region'].unique()
result = {}
for r in regions:
    mask = df['region'] == r
    result[r] = df.loc[mask, 'sales'].sum()
print(result)
# {'North': 250, 'South': 200, 'East': 300}

The manual approach breaks down for multiple grouping columns, complex aggregations, and large datasets. It's also hard to read and maintain. Groupby operations solve this by providing a clean, declarative syntax that is both expressive and performant.

Core concept / mental model

Think of groupby as split-apply-combine — a three-step process:

1. Split: Break the DataFrame into groups based on one or more columns.
2. Apply: Run a function (e.g., sum, mean, count, custom) on each group independently.
3. Combine: Stitch the results into a new DataFrame or Series.

Definitions

• Grouping column(s): The column(s) that define the groups (e.g., region, product, date).
• Aggregation: Applying a function that reduces each group to a single value (e.g., sum, mean, max).
• Transformation: Applying a function that returns a value of the same shape as the group (e.g., z-score, fillna).
• Filtration: Dropping groups based on a condition (e.g., keep groups with at least 3 rows).

Diagram-in-words

Imagine your data is a deck of cards, and you want the average value per suit. You split the deck into four piles by suit, compute the mean for each pile, then lay the four means side by side. groupby does exactly that — but with unlimited piles and sophisticated functions.

How it works step by step

Step 1: Import pandas and load your data

Always start with import pandas as pd. Load your data or create a sample DataFrame for practice.

Step 2: Call groupby() on the DataFrame

df.groupby('column') returns a GroupBy object — this is not the final result. It's a lazy intermediary that waits for you to specify an operation. You must follow it with an aggregation, transformation, or filter.

Step 3: Apply an aggregation

Chain a method like .sum(), .mean(), .count(), or .agg() to compute group-wise summaries. The default behavior is to apply the function to all numeric columns, but you can target specific columns.

Example sequence

# Step-by-step groupby
grouped = df.groupby('region')  # split
total = grouped['sales'].sum()   # apply + combine
print(total)

Pro tip: The GroupBy object is lazily evaluated. It doesn't compute anything until you trigger a reduction, so you can safely build complex pipelines without performance penalties.

Hands-on walkthrough

Let's work with a real-world-style dataset: sales records with region, product, and revenue.

Basic aggregation

import pandas as pd

df = pd.DataFrame({
    'region': ['North', 'South', 'North', 'East', 'South', 'East'],
    'product': ['Widget', 'Gadget', 'Widget', 'Gadget', 'Widget', 'Gadget'],
    'revenue': [100, 200, 150, 300, 250, 400],
    'units': [10, 20, 15, 30, 25, 40]
})

# Total revenue per region
region_total = df.groupby('region')['revenue'].sum()
print(region_total)
# region
# East     700
# North    250
# South    450
# Name: revenue, dtype: int64

# Average units per product
product_avg = df.groupby('product')['units'].mean()
print(product_avg)
# product
# Gadget    30.0
# Widget    16.667
# Name: units, dtype: int64

Multiple aggregations with agg

When you need different functions for different columns or a list of stats per column, use .agg():

# Multiple stats per group
summary = df.groupby('region').agg({
    'revenue': ['sum', 'mean', 'max'],
    'units': 'sum'
})
print(summary)

Group by multiple columns

You can group by several columns at once — the result has a MultiIndex:

multi = df.groupby(['region', 'product']).sum()
print(multi)
#                     revenue  units
# region product                   
# East   Gadget          700     70
# North  Widget          250     25
# South  Gadget          200     20
#        Widget          250     25

Using as_index=False for a clean result

By default, the grouping columns become the index. To keep them as regular columns, use as_index=False:

clean = df.groupby('region', as_index=False)['revenue'].sum()
print(clean)
#   region  revenue
# 0   East      700
# 1  North      250
# 2  South      450

Custom aggregation functions

You can pass any function that takes a Series and returns a scalar:

def range_value(s):
    return s.max() - s.min()

result = df.groupby('region')['revenue'].agg(range_value)
print(result)
# region
# East     100
# North     50
# South     50
# Name: revenue, dtype: int64

Pro tip: For performance, favor built-in methods (sum, mean, count, median) over custom Python functions when possible — pandas is heavily optimized for them.

Compare options / when to choose what

When to choose what:

• Use groupby when you need fine control over group-wise operations (custom functions, multiple aggregations).
• Use pivot_table when you want a spreadsheet-like summary with rows and columns.
• Use crosstab when you specifically need a frequency table of two categorical variables.

Troubleshooting & edge cases

Grouping by a column with missing values

By default, NaN values in the grouping column are excluded. To include them as a group, use dropna=False:

df = pd.DataFrame({'city': ['NY', None, 'LA'], 'sales': [100, 200, 150]})
result = df.groupby('city', dropna=False)['sales'].sum()
print(result)
# city
# LA     150
# NY     100
# NaN    200
# Name: sales, dtype: int64

Getting an empty or unexpected result

If your result is empty, check that the grouping column exists and has non-null values. Also, ensure the grouping column is of the right type (e.g., string vs. datetime).

Performance issues on large data

groupby is fast, but if you're using custom Python functions, consider vectorizing or using pandas.transform for parallel-friendly operations. Also, avoid grouping on high-cardinality columns (e.g., unique IDs) unless necessary.

KeyError on column name

If you get a KeyError, the column name might contain spaces or different casing. Double-check the DataFrame columns with df.columns.

Unexpected aggregation on non-numeric columns

When you call .sum() on a groupby without selecting a column, pandas tries to sum numeric columns only and may exclude others silently. Always select the columns you want to aggregate explicitly.

What you learned & what's next

You've learned the core of group data with groupby operations: the split-apply-combine mental model, how to create GroupBy objects, apply aggregations, use multiple columns, custom functions, and avoid common pitfalls. These skills are essential for data summarization and will appear in nearly every data science project.

Next lesson in this track will take you into merging and joining datasets — how to combine multiple DataFrames using merge and concat. That's a natural partner to groupby: you group what you have, and combine what you need. Keep practicing with real datasets to solidify your understanding.

Pro tip: Master groupby and you'll save yourself hours of manual data wrangling. It's one of the most powerful verbs in the pandas language.