Encode Categorical Features

Your dataset is full of strings — country names, product categories, device types — and your machine learning model refuses to touch them. You’re staring at a TypeError or a model that silently treats every category as an ordered integer. This is the moment every data scientist meets: you have to encode categorical features with scikit-learn before any algorithm can learn from them. In this lesson, you'll master the two most powerful tools — OneHotEncoder and OrdinalEncoder — understand when to use which, and walk through hands-on examples that you can adapt to your own projects immediately.

The problem this lesson solves

Most machine learning algorithms, from linear regression to neural networks, perform mathematical operations like addition and multiplication on their inputs. They simply cannot compute on raw text like "red" or "compass". If you try to feed a pandas DataFrame with object columns into LinearRegression, you’ll get a clear but unhelpful failure:

from sklearn.linear_model import LinearRegression
import pandas as pd

X = pd.DataFrame({"color": ["red", "blue", "green"], "size": [1, 2, 3]})
y = [10, 20, 15]

lr = LinearRegression()
lr.fit(X, y)  # ⚠️ ValueError: could not convert string to float: 'red'

Why this hurts: Most models expect a 2D numeric array. Strings are invisible to them. You must transform each categorical column into a numerical representation that preserves the information the model needs.

But there’s a second, more subtle problem: even if you manually map categories to integers, you might unknowingly imply an order that doesn't exist. If you encode ["small", "medium", "large"] as [0, 1, 2], that’s fine for size. But if you encode ["cat", "dog", "bird"] as [0, 1, 2], you’re telling the model that dog is greater than cat — a dangerous assumption that will degrade predictions.

This lesson gives you a reliable, scikit-learn-native workflow to encode any categorical feature correctly, avoiding both failures.

Core concept / mental model

Think of categorical encoding as translation from human language to machine language. You have two main dialects:

• Ordinal encoding — assigns a ranked integer to each category. Use it for ordinal categories that have a natural, meaningful order (e.g., low, medium, high).
• One-hot encoding — creates a binary column for each category, with a 1 in the column corresponding to the category of each row. Use it for nominal categories where no order exists (e.g., animal, city, color).

Scikit-learn provides two clean, pipeline-friendly transformers:

• OrdinalEncoder — maps each unique category to an integer (0, 1, 2, …).
• OneHotEncoder — expands the column into as many binary columns as there are categories.

Both are designed to integrate with ColumnTransformer so you can encode multiple columns, mixed types, and then feed the result straight into a model.

A key intuition: one-hot encoding increases dimensionality but removes any false order. Ordinal encoding keeps dimensions low but imposes an order that may not exist. Your choice matters — the wrong one can distort your model’s learning.

How it works step by step

Here’s a repeatable workflow to encode any categorical feature with scikit-learn:

1. Inspect your data – Identify which columns are categorical (dtype object or category) and decide whether each is ordinal or nominal.
2. Choose the encoder – For nominal: OneHotEncoder. For ordinal: OrdinalEncoder (with an explicit category order if possible).
3. Fit the encoder on training data only – This captures the categories that exist in training. Never fit on the whole dataset to avoid data leakage.
4. Transform training and test data – Apply the same fitted encoder to both, ensuring consistent encoding.
5. Combine with ColumnTransformer – For mixed datasets, encode categorical columns while passing numeric columns through unchanged.
6. Pipe into a model – Plug the transformer into a Pipeline for clean, reproducible preprocessing.

Hands-on walkthrough

1. OneHotEncoder for nominal categories

Let’s encode a color column. We’ll use OneHotEncoder with default settings, which returns a sparse matrix by default — use sparse_output=False for a dense array.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

df = pd.DataFrame({
    "color": ["red", "blue", "green", "blue"],
    "size": [1, 2, 3, 4]
})

encoder = OneHotEncoder(sparse_output=False)
encoded = encoder.fit_transform(df[["color"]])

encoded_df = pd.DataFrame(encoded, columns=encoder.get_feature_names_out(["color"]))
print(encoded_df)

Output:

   color_blue  color_green  color_red
0         0.0          0.0        1.0
1         1.0          0.0        0.0
2         0.0          1.0        0.0
3         1.0          0.0        0.0

2. OrdinalEncoder for ordinal categories

For sizes, we specify the order so that small=0, medium=1, large=2.

from sklearn.preprocessing import OrdinalEncoder
import pandas as pd

sizes = pd.DataFrame({"size": ["small", "large", "medium", "small"]})

encoder = OrdinalEncoder(categories=[["small", "medium", "large"]])  # order matters!
encoded = encoder.fit_transform(sizes)

print(pd.DataFrame(encoded, columns=["size_encoded"]))

Output:

   size_encoded
0           0.0
1           2.0
2           1.0
3           0.0

3. Combined pipeline with ColumnTransformer

Now the real-world case: a dataset with a mix of numeric and categorical features. Use ColumnTransformer to encode only the categorical columns and pass numeric columns unchanged.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LinearRegression
import pandas as pd

# Sample data
df = pd.DataFrame({
    "size_sqft": [1000, 1500, 1200, 1800],
    "city": ["NYC", "SF", "Austin", "NYC"],
    "rooms": [2, 3, 2, 4]
})
y = [300_000, 400_000, 250_000, 500_000]

preprocessor = ColumnTransformer(
    transformers=[
        ("city_enc", OneHotEncoder(), ["city"]),
        ("num", StandardScaler(), ["size_sqft", "rooms"])
    ]
)

pipe = Pipeline(steps=[
    ("prep", preprocessor),
    ("reg", LinearRegression())
])

pipe.fit(df, y)
print("Predictions:", pipe.predict(df))

Output (values may vary due to scaling):

Predictions: [300000.0 400000.0 250000.0 500000.0]

The pipeline fits perfectly (overfit by design) to show the flow works end-to-end.

Compare options / when to choose what

The table below summarizes the key trade-offs:

The golden rule: if order isn't real, use one-hot; if it is, use ordinal. For high-cardinality categories (e.g., 10,000+ unique cities), consider alternatives like target encoding or the Sparse output of OneHotEncoder, but those are advance topics.

Troubleshooting & edge cases

• New categories in test data that weren’t in training – OneHotEncoder will raise an error if it sees an unseen category. Set handle_unknown='ignore' to give all unseen values zero across all columns.
• Sparse matrix confusion – Default OneHotEncoder returns a sparse matrix. If you try to concatenate directly with pd.concat, it fails. Use sparse_output=False for dense, or convert with .toarray().
• Fitting on the whole dataset before splitting – Always fit on the training set only. Otherwise you leak information about category frequencies from the test set.
• Misusing LabelEncoder on features – LabelEncoder is meant for the target variable in classification. For features, use OrdinalEncoder — LabelEncoder will treat your feature column as the target and corrupt it.
• Forgetting to handle NaN – Encoders, by default, fail on missing values. Either drop rows, impute, or use np.nan in categories (with handle_unknown='ignore') to allow NaN encoding.

What you learned & what's next

You now can encode categorical features with scikit-learn. You learned to distinguish ordinal from nominal, to use OneHotEncoder and OrdinalEncoder correctly, and to embed them in a Pipeline with ColumnTransformer. This is a foundational preprocessing skill that appears in nearly every applied AI project.

Next up in your Applied AI engineering path: feature scaling and selection — preparing your numeric features just as rigorously. With these combined, your models will start with a clean, meaningful feature matrix every time.

Now, go ahead and encode — your models will thank you.