Categorize Data with Categorical Types

You’ve mastered DataFrames, slicing, and aggregation, but every time you run df.groupby('status').size(), you feel that nagging inefficiency. Maybe you’ve even noticed that a column of repeated strings — like 'active', 'inactive', 'pending' — is eating up memory and slowing down your filters. The fix is hiding in plain sight: categorical types. In this lesson, you’ll learn how to categorize data with categorical types, a pandas feature that turns messy, repetitive text into a blazing-fast, memory-savvy representation. By the end, you’ll not only save RAM but also unlock powerful ordering and plotting behaviors that plain object columns can’t give you.

The problem this lesson solves

Picture an e-commerce dataset with two million orders. The status column holds strings like 'delivered', 'shipped', 'cancelled'. Each string is stored as a full Python object, which means pandas has to keep two million references to identical text. That’s a huge waste of memory — and it slows down every filter, sort, and groupby operation. Worse, when you try to sort by 'delivered', 'shipped', 'cancelled', you get alphabetical order, not your business logic order (e.g., cancelled → shipped → delivered).

The core problem is that plain object columns treat every value as a unique piece of text. There’s no awareness that these values belong to a fixed set of categories. You might also face inconsistent labels — 'Delivered', 'delivered ' — that make grouping produce stray categories. This lack of structure makes your data slower, messier, and harder to reason about.

Consider this: in a real-world analytics pipeline, you might run the same query hundreds of times. The performance penalty compounds. You need a way to declare, once, “these are the only possible values, and they have a specific order.” That’s exactly what categorical types provide.

Core concept / mental model

Think of a categorical type like a dictionary of unique values plus a compact list of integer codes. For the column ['red', 'blue', 'red', 'green'], pandas stores:

• Categories: ['red', 'blue', 'green'] (unique values, optional order)
• Codes: [0, 1, 0, 2] (integer references)

Instead of storing four full strings, pandas stores four tiny integers and one set of unique labels. When you display the series, it still looks like strings — but underneath it’s numbers. This is the same idea as factor levels in R, enums in programming languages, or lookup tables in a database.

Mental model: Think of a categorical column as a foreign key to a small lookup table of valid labels. The DataFrame holds the foreign key (integer codes); the category dictionary holds the human-readable text. This separation is what gives you both speed and clarity.

This model also explains why categoricals can be ordered. Because categories have a defined sequence, matplotlib and pandas can plot bars in that order, and you can sort rows accordingly. It’s not just about memory — it’s about embedding metadata (the allowed values and order) into your data structure.

How it works step by step

Converting a column to a categorical type is easy, but it’s worth understanding the anatomy. Here’s the step-by-step process:

1. Identify columns that hold a limited set of repeating values. Look for strings or integers that repeat heavily — statuses, categories, regions, or star ratings.
2. Create the categorical using pd.Categorical(values, categories=[...], ordered=True/False), or convert an existing Series with .astype('category').
3. Define categories explicitly when you care about order or want to catch unexpected values (e.g., a typo like 'delivred' will raise an error if not in categories).
4. Use the categorical in aggregations, filters, and visualizations. Sorting now follows the category order, not lexicographic order.
5. Optimize further by setting the categorical as the index or using it with groupby(observed=True) to avoid empty groups.

Let’s walk through a small example:

import pandas as pd

# Sample data with repeated strings
orders = pd.DataFrame({
    'status': ['shipped', 'delivered', 'cancelled', 'delivered', 'shipped', 'delivered'],
    'amount': [120, 85, 200, 95, 300, 110]
})

# Convert to categorical with an explicit order
status_cat = pd.Categorical(
    orders['status'],
    categories=['cancelled', 'shipped', 'delivered'],
    ordered=True
)

# Assign back to the DataFrame
orders['status'] = status_cat

print(orders.dtypes)
# status    category
# amount       int64

dtype: object

Now when you sort, the order follows your business logic:

print(orders.sort_values('status'))

  status  amount
2  cancelled     200
0    shipped     120
4    shipped     300
1  delivered      85
3  delivered      95
5  delivered     110

Notice that cancelled comes first, then shipped, then delivered. That’s the power of an ordered categorical.

When to set ordered=True

Use ordered=True when the categories have a natural or business hierarchy — e.g., education level, rating, or lifecycle stages. Use ordered=False (the default) when order doesn’t matter, like colors or product types. Ordered categoricals also enable comparison operators like <, >, and min()/max().

Hands-on walkthrough

Let’s build a complete exercise that mirrors a real-world analytics scenario. We’ll load a dataset of customer feedback with a satisfaction column, convert it to an ordered categorical, and then analyze trends.

Step 1: Create a synthetic dataset

import pandas as pd

# Simulate survey responses
responses = pd.DataFrame({
    'customer_id': range(1, 11),
    'satisfaction': ['low', 'high', 'medium', 'low', 'high', 'medium', 'medium', 'low', 'high', 'low'],
    'region': ['east', 'west', 'east', 'west', 'east', 'east', 'west', 'west', 'east', 'west']
})

# Convert to categorical with a defined order
responses['satisfaction'] = pd.Categorical(
    responses['satisfaction'],
    categories=['low', 'medium', 'high'],
    ordered=True
)

print(responses.dtypes)
print()
print(responses['satisfaction'])

customer_id       int64
satisfaction    category
region           object
dtype: object

0      low
1     high
2   medium
3      low
4     high
5   medium
6   medium
7      low
8     high
9      low
Name: satisfaction, dtype: category
Categories (3, object): ['low' < 'medium' < 'high']

Notice how the output shows Categories (3, object): ['low' < 'medium' < 'high'] — that’s the machine-readable metadata.

Step 2: Use the categorical in analysis

Now we can count, sort, and even compare values:

# Count by category, in order
counts = responses['satisfaction'].value_counts()
print("Counts per level:")
print(counts)

# Filter for high satisfaction
high = responses[responses['satisfaction'] == 'high']
print("\nHigh satisfaction customers:")
print(high['customer_id'].tolist())

# Compare categories (only possible with ordered=True)
print("\nIs 'medium' greater than 'low'?", responses['satisfaction'][0] < 'medium')

Counts per level:
low       4
medium    3
high      3
Name: satisfaction, dtype: int64

High satisfaction customers:
[2, 5, 9]

Is 'medium' greater than 'low'? True

The value_counts() automatically respects the category order, so the output lists low, medium, high instead of alphabetical or arbitrary order.

Step 3: Memory footprint check

Let’s measure the memory savings with a larger sample:

import numpy as np

# 100,000 random strings from a small set
np.random.seed(42)
statuses = np.random.choice(['active', 'inactive', 'pending'], size=100_000)

df = pd.DataFrame({'status': statuses})
print("Memory before (object):", df.memory_usage(deep=True)['status'])

df['status'] = df['status'].astype('category')
print("Memory after (category):", df.memory_usage(deep=True)['status'])

Memory before (object): 8000000 bytes
Memory after (category): 100112 bytes

That’s an 80× reduction! Instead of storing 8 MB of repeated strings, pandas stores a small code vector plus a tiny category dictionary. In big datasets, this difference can be the difference between “runs out of memory” and “completes instantly.”

Step 4: Group and visualize

Categoricals shine in groupby because empty groups can be suppressed with observed=True (useful when your categories are explicit). Let’s glance at a quick plot:

import matplotlib.pyplot as plt

# Group by region and satisfaction
pivot = responses.groupby(['region', 'satisfaction'], observed=False).size().unstack()
print(pivot)

# Plot as a bar chart — order follows categories, not alphabet
pivot.plot(kind='bar', title='Satisfaction by Region')
plt.tight_layout()
plt.show()

satisfaction  low  medium  high
region                        
east            2       2     2
west            2       1     1

The plot will show regions on the x-axis and satisfaction levels as grouped bars, appearing in low → medium → high order — making the pattern instantly readable without extra sorting.

Compare options / when to choose what

You have several ways to handle repeated categorical-like values in pandas. Here’s a quick comparison:

For “categorize data with categorical types,” the pd.Categorical explicit approach is the gold standard. It gives you memory savings, ordering, and validation — all at once. Use astype('category') as a quick win when you don’t care about order; switch to explicit pd.Categorical when the column has a natural hierarchy or you are building a reusable pipeline.

Pro tip: If you have a column that is 1% numeric codes and 99% labels, use a dictionary to map codes to labels, then convert to categorical. This gives you a lightweight, human-readable column that still packs the speed of integers.

Troubleshooting & edge cases

Even with categoricals, things can go wrong. Here are common issues and fixes:

ValueError: Categorical categories must be unique

This happens when you pass duplicate categories, e.g., categories=['low', 'high', 'low']. Pandas requires each category to be unique. Fix by deduplicating your list or using pd.unique().

New values appear after conversion

If you forget to include a value in the categories argument, it won’t be in the categorical — it becomes NaN. For example, if your data has 'unknown' but you only listed ['low', 'medium', 'high'], 'unknown' will be missing. Options: add it to categories, or clean your data first.

Sorting doesn’t respect business order

You might see alphabetical order if you used astype('category') without specifying ordered=True. Remember: astype('category') creates an unordered categorical. Use pd.Categorical(..., ordered=True) to enforce order.

groupby produces many empty groups

If you define categories that don’t appear in the data, groupby will show them as empty groups. Use observed=True to only show observed combinations, or observed=False to keep all (default). This is especially relevant with multiple categorical columns.

Memory still looks high

If your column has many unique values (e.g., 90% of rows are unique), categoricals may not save memory — sometimes they even take more. Check df['col'].nunique() before converting. Only convert when the number of categories is small relative to the row count.

ValueError: Cannot set a Categorical with another, without identical categories

This occurs when you try to assign a categorical with different categories to an existing categorical column. To safely merge, use pd.Categorical.concat(..., categories=...) or update categories with cat.set_categories().

Pro tip: Always verify your categories with df['col'].cat.categories after conversion. It’s a quick sanity check that your data is exactly as you expect.

What you learned & what's next

You’ve learned the core idea behind categorize data with categorical types: replacing repeated strings with a compact integer-code representation. You now know how to create categoricals with pd.Categorical and astype('category'), why ordered categoricals are powerful for sorting and comparisons, and how to measure memory savings. You also saw practical troubleshooting for common pitfalls like duplicate categories, unexpected missing values, and sorting order — and you can confidently apply this to your own datasets to make them faster and more meaningful.

Next stop: now that your columns are properly typed, it’s time to combine them with other powerful pandas features. Up next, you’ll learn how to slice and dice your categorical data with advanced groupby and pivot operations — turning this new type system into analytical firepower. Keep an eye on observed=True and you’ll be a master of high-performance aggregation in no time.