Group Data with Groupby

Every dataset tells a story, but the raw rows rarely speak clearly. When you need to answer questions like "What is the average revenue per region?" or "How many orders did each customer place last month?", you could write endless loops and conditionals — but that's slow, error-prone, and hard to read. The pandas groupby operation is the single most powerful tool in your data science toolkit for splitting data into groups, applying a function to each group, and combining the results. This lesson makes you fluent in the split-apply-combine pattern, turning messy tables into clean, actionable insights in a few lines of Python.

The problem this lesson solves

Before you learn groupby, tackling aggregated questions about your data means reaching for manual loops, or painstakingly filtering the DataFrame for each category. Consider a sales dataset with thousands of rows. You need the total sales per product category. Without groupby, you might loop over unique categories, filter the DataFrame each time, and sum a column — a verbose and brittle approach:

import pandas as pd

df = pd.DataFrame({
    'category': ['Electronics', 'Clothing', 'Electronics', 'Clothing', 'Home'],
    'sales': [100, 50, 75, 60, 90]
})

categories = df['category'].unique()
result = {}
for cat in categories:
    result[cat] = df[df['category'] == cat]['sales'].sum()
print(result)
# {'Electronics': 175, 'Clothing': 110, 'Home': 90}

This works for one column, but imagine doing it for multiple columns, custom functions, or needing percentiles and counts. The code balloons and becomes a maintenance nightmare. The core problem is that you're writing imperative plumbing instead of declarative analysis. groupby solves this by giving you a concise, expressive, and fast way to apply any operation to groups of rows — the cornerstone of exploratory data analysis.

Core concept / mental model

Think of groupby as a split-apply-combine machine, a concept popularized by Hadley Wickham in R. The mental model has three stages:

1. Split: The DataFrame is divided into groups based on the values in one or more key columns. Each group is a subset of rows sharing the same key value(s).
2. Apply: A function (like sum, mean, count, or a custom function) is applied independently to each group.
3. Combine: The individual results are brought back together into a single DataFrame or Series.

Visualize it like sorting a deck of cards by suit. You split the deck into four piles (split), count or rank each pile (apply), and then stack the summaries together (combine). The original rows stay intact; only the summary is combined.

In pandas, the groupby object is lazy — it merely stores the grouping logic until you call an aggregation method like .sum() or .agg(). This is a powerful idea: you define the grouping once and can run many different analyses on the same groups.

How it works step by step

Let's break down the groupby workflow into a sequence you can apply to any dataset:

1. Identify your grouping key(s) — the column(s) that define your groups. This is usually a categorical column like region, product_type, or customer_id.
2. Select the column(s) to aggregate — optionally specify a single column (e.g., df['sales']). If you omit this, all remaining numeric columns are aggregated.
3. Call groupby() with your key(s) — pandas creates the DataFrameGroupBy object.
4. Apply an aggregation method — .sum(), .mean(), .count(), .agg(), or .apply(). The function transforms each group and returns a result.
5. Inspect and use the result — a new DataFrame or Series with the group keys as the index (or columns, if you use as_index=False).

The syntax is elegantly simple:

agg_result = df.groupby('category')['sales'].sum()

Here is the cause-and-effect chain: splitting produces groups → applying sums each group's sales → combining returns a Series indexed by category, sorted by default.

Now, let's extend to multiple keys. You can group by two columns by passing a list: df.groupby(['category', 'region'])['sales'].mean(). This creates a hierarchical index (MultiIndex). To keep the result flat, use as_index=False — a common trick to convert the group keys back into regular columns.

You can also group by any column you like, not just categorical ones. Group by a numeric column (e.g., age) to get per-age summaries. But beware — grouping by a high-cardinality numeric column creates many groups; binning (e.g., pd.cut) is often preferred.

Hands-on walkthrough

Let’s bring it all together with a realistic example. We'll use a small sales DataFrame and perform a series of common groupby operations.

import pandas as pd

# Sample sales data
df = pd.DataFrame({
    'region': ['North', 'South', 'North', 'West', 'South', 'West', 'North'],
    'product': ['A', 'B', 'A', 'B', 'A', 'B', 'A'],
    'sales': [100, 150, 200, 120, 80, 90, 170],
    'units': [2, 3, 4, 2, 1, 3, 3]
})

# Group by region and get total sales and average units
summary = df.groupby('region').agg({'sales': 'sum', 'units': 'mean'})
print(summary)
#         sales     units
# region
# North     470  3.000000
# South     230  2.000000
# West      210  2.500000

Notice how the result is indexed by region. If you want the region as a regular column, add as_index=False:

summary_flat = df.groupby('region', as_index=False)['sales'].sum()
print(summary_flat)
#   region  sales
# 0  North    470
# 1  South    230
# 2  West     210

Custom aggregation. Need more than a simple sum? Use .agg() with multiple functions:

stats = df.groupby('product')['sales'].agg(['sum', 'mean', 'count'])
print(stats)
#         sum  mean  count
# product
# A        550   137.5      4
# B        360   120.0      3

Group by multiple columns — this creates a grouped DataFrame with hierarchical index:

multi = df.groupby(['region', 'product'])['sales'].sum()
print(multi)
# region  product
# North   A          470
# South   B          150
#         A           80
# West    B          210
# Name: sales, dtype: int64

Iterating over groups — sometimes you need each group as a separate DataFrame:

for name, group in df.groupby('region'):
    print(f"Region: {name}, total units: {group['units'].sum()}")
# Region: North, total units: 9
# Region: South, total units: 6
# Region: West, total units: 6

The apply method is the most flexible — it passes each group as a DataFrame to your function:

def custom(group):
    return group['sales'].sum() * 2

print(df.groupby('region').apply(custom))
# region
# North    940
# South    460
# West     420
# dtype: int64

Pro tip: Use .size() to count rows per group — it's different from .count(), which ignores NaN values. .size() counts every row, including those with missing data.

Compare options / when to choose what

groupby offers several ways to achieve similar results. The table below compares the most common methods to help you choose the right tool.

When to choose what? Use .agg() when you need multiple statistics at once. Use .apply() when your logic is complex and can't be expressed as a simple aggregation. Use .transform() when you need to create a new column with group-level statistics aligned to the original rows. Use pivot_table when you want a cross-tabulated view (categories as rows, another category as columns).

Alternative: groupby vs. pivot_table for grouped summaries. Both handle split-apply-combine, but pivot_table reshapes the data into a wide format, which is useful for comparison across two dimensions. groupby keeps a long format, which is often better for further computation.

Alternative: vectorized operations with NumPy or pandas without groupby — such as df[df['cat'] == 'A'] — are only suitable for a fixed number of known groups. groupby scales automatically to any number of groups, making it the standard for exploratory analysis.

Troubleshooting & edge cases

Grouping by a column with missing values: By default, rows with None or NaN in the grouping column are dropped. To keep them as their own group, set dropna=False:

df = pd.DataFrame({'cat': ['A', None, 'A'], 'val': [1, 2, 3]})
print(df.groupby('cat', dropna=False)['val'].sum())
# cat
# A     4.0
# NaN   2.0
# Name: val, dtype: float64

Using .size() vs .count(): If you expect the number of rows per group, use .size() — it counts all rows including those with NaN in any column. .count() only counts non-null values in the selected column, which can be lower if there are missing values. Getting the wrong one can silently skew your analysis.

TypeError: 'DataFrameGroupBy' object is not callable — This happens if you forget parentheses around the aggregation method, like df.groupby('cat').sum instead of .sum(). Always call the method.

Surprise with as_index: When you group by a column and use as_index=True (default), the group key becomes the index. If you later try to reset it, use .reset_index(). If you'd rather have the key as a column from the start, use as_index=False.

Aggregating non-numeric columns: If you try to .sum() a group containing string columns, pandas will concatenate the strings (the + operator) or raise an error. Be explicit about which columns you aggregate.

Empty groups: When filtering after grouping, you might accidentally create a group with no data. groupby handles this gracefully, but your aggregation may still produce NaN for that group. Decide how to handle missing summaries — sometimes .fillna(0) is appropriate.

What you learned & what's next

You now have a solid understanding of group data with groupby operations in pandas. You learned the split-apply-combine pattern — the mental model that powers every groupby — and you applied it in hands-on examples to compute sums, means, counts, and custom aggregations. You can group by single or multiple columns, iterate over groups, use apply and transform, and you know when to reach for pivot_table instead. You also know how to avoid common pitfalls like the wrong count method and the missing-keys trap.

With this skill, you can summarize, explore, and compare subsets of data at scale — a critical step in any data science workflow. You're now one step closer to telling compelling data stories.

Next step: In the next lesson, you'll learn how to merge and join DataFrames — the essential skill for combining multiple tables to enrich your analysis. Mastering groupby plus join operations gives you the two pillars of data wrangling that will prepare you for advanced visualization and modeling.