Encode categorical variables

Your pandas DataFrame is full of columns like 'color', 'city', or 'category' — but your machine learning model only understands numbers. Feed it strings and you'll hit a ValueError or, worse, silently get nonsense predictions. That's the pain: categorical variables are everywhere in real-world data, and encoding them correctly is the difference between a model that learns and one that fails. In this lesson, you'll master the art of encoding categorical variables with pandas — the clean, explicit way to turn labels into numbers you can feed into any algorithm.

The problem this lesson solves

Most machine learning libraries — scikit-learn, statsmodels, TensorFlow — expect numeric input. A column of strings like ['red', 'blue', 'green'] cannot be subtracted, multiplied, or fed into a linear regression. The moment you try, you'll see errors or, if you're using a library that silently converts, you'll get rankings that make no semantic sense.

But it's not just about making strings numeric. It's about how you do it. A naive approach — assigning red=1, blue=2, green=3 — implies an order that doesn't exist. Your model might learn that green is "bigger" than blue, which is pure nonsense. Encoding categorical variables with pandas is both a mechanical task (transforming data types) and a semantic task (preserving the meaning of your categories).

You'll face this in real projects constantly: survey responses ('Strongly agree'), product categories ('Electronics'), geographic regions ('North', 'South'), and even dates treated as categories. Without a solid encoding strategy, your model's performance — and your credibility as a data scientist — suffers.

Core concept / mental model

Think of categorical variables as labels in a box, and encoding as the act of translating those labels into a language numbers can understand. But not all translations are equal. You need to decide what the labels mean:

• Is there a natural order? ('Low' < 'Medium' < 'High') → Ordinal encoding
• Is there no order at all? ('Red', 'Green', 'Blue') → One-hot encoding (or dummy encoding)
• Is the category a label rather than a feature? If it's your target column, you might use label encoding for classification.

Think of it this way: if the categories have a ranking, use ordinal encoding. If you'd be embarrassed to say "green > red", use one-hot. It's a simple test that prevents 90% of modeling mistakes.

Here's a mental diagram to hold onto:

Categorical column
      |
      ├── Nominal (no order) → one-hot encoding → multiple binary columns (0/1)
      |
      └── Ordinal (has order) → ordinal encoding → single column with integers (0, 1, 2...)

Pandas provides two core methods for this:

• pd.get_dummies() — creates one-hot encoded columns (0/1 for each category)
• pd.Categorical() with categories — supports ordinal encoding with controlled order

Plus, you have powerful options in sklearn's OrdinalEncoder and OneHotEncoder, which we'll compare later.

How it works step by step

Encoding categorical variables with pandas is a three-step process:

1. Identify your categorical columns — scan your DataFrame for columns with dtype object or category. Use df.select_dtypes(include=['object']) to find them.
2. Choose your encoding strategy — ask: Is the variable ordinal or nominal? Is it an input feature or the target? How many unique categories does it have? (A cardinality of 50+ might call for different handling.)
3. Apply the encoding — use pd.get_dummies() for one-hot, or pd.Categorical() / a mapping for ordinal. Then check your result for correctness.

Let's walk through each step with code.

First, create a sample DataFrame:

import pandas as pd

df = pd.DataFrame({
    'color': ['red', 'green', 'blue', 'green', 'red'],
    'size': ['S', 'M', 'L', 'M', 'L'],
    'price': [10, 15, 20, 15, 25]
})
print(df)

Output:

   color size  price
0    red    S     10
1  green    M     15
2   blue    L     20
3  green    M     15
4    red    L     25

Here, color is nominal, size is ordinal (S < M < L). price is numeric and untouched.

Hands-on walkthrough

One-hot encoding with pd.get_dummies()

The simplest way to encode nominal categories is one-hot encoding. For each unique category, you get a new binary column (0 or 1). Pandas makes this trivial:

# One-hot encode 'color' column
df_encoded = pd.get_dummies(df, columns=['color'])
print(df_encoded)

Output:

   size  price  color_blue  color_green  color_red
0    S     10           0            0          1
1    M     15           0            1          0
2    L     20           1            0          0
3    M     15           0            1          0
4    L     25           0            0          1

Notice that the original color column disappeared, replaced by three new columns. The size column remains as text because we didn't encode it — that's fine for now, but you'll likely want to encode it too.

Pro tip: Use pd.get_dummies(df, drop_first=True) to avoid the dummy variable trap (perfect multicollinearity). This drops the first category, leaving k-1 columns for k categories — enough to encode all information.

df_encoded_drop = pd.get_dummies(df, columns=['color'], drop_first=True)
print(df_encoded_drop)

Output:

   size  price  color_green  color_red
0    S     10            0          1
1    M     15            1          0
2    L     20            0          0
3    M     15            1          0
4    L     25            0          0

Now we have only two columns for three categories — green and red are explicit, blue is implied when both are 0.

Ordinal encoding for ordered categories

For size (S, M, L) we want to preserve the order. A simple mapping works:

size_map = {'S': 0, 'M': 1, 'L': 2}
df['size_ordinal'] = df['size'].map(size_map)
print(df[['size', 'size_ordinal']])

Output:

  size  size_ordinal
0    S             0
1    M             1
2    L             2
3    M             1
4    L             2

Alternatively, you can use pandas' Categorical dtype:

df['size_cat'] = pd.Categorical(df['size'], categories=['S', 'M', 'L'], ordered=True)
# This gives you an ordered categorical, which some models can handle directly.

But for most modeling libraries, you'll need integers, so the map approach is more practical.

Combining both encodings

Let's encode both columns in one go:

df_final = pd.get_dummies(df, columns=['color'], drop_first=True)
df_final['size_ordinal'] = df['size'].map({'S': 0, 'M': 1, 'L': 2})
df_final = df_final.drop(columns=['size'])
print(df_final)

Output:

   price  color_green  color_red  size_ordinal
0     10            0          1             0
1     15            1          0             1
2     20            0          0             2
3     15            1          0             1
4     25            0          0             2

Now your DataFrame is fully numeric and ready for any model.

Check: Every value in your encoded columns should be either 0, 1, or a small integer that reflects your mapping. If you see NaN, you probably had categories in your test data that weren't in the training data — more on that in troubleshooting.

Compare options / when to choose what

When to choose what:

• If your category is nominal and has fewer than ~20 unique values, one-hot encoding is the default choice. It's interpretable and safe.
• If your category is ordinal, use a mapping like {'S':0, 'M':1, 'L':2} or pd.Categorical for that column.
• For high-cardinality nominal features (e.g., 1,000 unique cities), one-hot blows up. Consider target encoding or frequency encoding — beyond pandas' scope, but keep in mind (we'll mention them in variations).
• For your target column in classification, you can use LabelEncoder because the target column doesn't need to be interpretable by a linear model — it just needs to be numeric for fit().

Variations you might encounter in the wild:

• pd.get_dummies creates column names with prefixes (color_red) — you can control with prefix argument.
• sklearn's OneHotEncoder returns a sparse matrix, which is memory-efficient for large datasets; you can convert to DataFrame with toarray().
• Target encoding: replace each category with the mean of the target variable — powerful but must be done carefully to avoid leakage.

Troubleshooting & edge cases

1. ValueError: could not convert string to float

You forgot to encode a categorical column before feeding it to your model. Solution: check df.dtypes and encode object columns.

2. NaN appears after encoding

If your test data contains a category that wasn't in training, pd.get_dummies will just not create a column for it, resulting in a NaN when you try to align DataFrames. Solution: fit your encoder on training data only and use pd.get_dummies(train, columns=['color']) and pd.get_dummies(test, columns=['color']) separately, then align with reindex.

# Align test data to training columns
test_encoded = pd.get_dummies(test, columns=['color'])
test_encoded = test_encoded.reindex(columns=train_encoded.columns, fill_value=0)

3. Dummy variable trap

If you keep all one-hot columns for a category, many models (especially linear regression) suffer from multicollinearity. Always use drop_first=True or drop one column manually.

4. Memory blow-up with high cardinality

For a column with 1,000 unique values, one-hot creates 1,000 columns. This can eat gigabytes. Solutions: drop rare categories (e.g., keep top 20 based on frequency) before encoding, or use a different strategy like target encoding.

5. Ordinal encoding on nominal data

If you encode 'red', 'green', 'blue' as 1,2,3, you're implying blue > green. This can mislead tree-based models too. Always ask: "Does the order make sense?"

What you learned & what's next

You can now encode categorical variables with pandas confidently. You learned:

• How to identify categorical columns with df.select_dtypes()
• How to apply one-hot encoding with pd.get_dummies(), including drop_first to avoid multicollinearity
• How to apply ordinal encoding with a simple mapping or pd.Categorical
• How to choose between encoding methods based on ordinal vs nominal and cardinality
• How to handle common pitfalls like NaN in test data and dummy variable traps

The next lesson in this track will likely cover feature scaling — because once your categories are numbers, you'll want to bring all features onto a similar scale for models like SVM or K-means. You'll build on the same pipeline: clean data → encode → scale → train.

Remember: encoding is not a one-size-fits-all decision. Revisit your choice when your model underperforms — sometimes the encoding is the silent killer. Now go encode something!