Boolean Indexing in Python

You’ve got a DataFrame with thousands of rows, and you need just the ones where sales spiked, temperatures dropped, or users churned. Manually scanning or writing clunky loops is slow, error-prone, and doesn't scale. Boolean indexing — passing a Series of True/False values to select rows — is the clean, vectorized, and Pythonic way to filter your data in pandas and NumPy. It’s the single most important skill for everyday data wrangling, and once it clicks, you’ll never go back to for loops.

The problem this lesson solves

Filtering rows is the bread-and-butter of data analysis. Whether you're cleaning data, exploring patterns, or preparing training sets, you constantly need to answer questions like:

• Which products sold more than 100 units?
• Which customers are from Europe and churned?
• Which readings are above the 95th percentile?

Without a solid method, developers often fall back on verbose loops or nested if statements that are hard to read, slow on large datasets, and bug-prone. Boolean indexing solves this by letting you express your filter as a condition that evaluates to a boolean array, then using that array to select rows directly. The result is concise, readable, and fast because it leverages vectorized operations under the hood.

Why care now? As your datasets grow beyond a few hundred rows, manual inspection becomes impossible. Boolean indexing is the foundational tool you’ll use in dozens of later lessons — from grouping and reshaping to visualization and modeling.

Core concept / mental model

Think of boolean indexing as a mask or a filter. Imagine you have a row of physical filters — each filter is either open (True) or closed (False). When you pass a sequence of filters to your data, only the items with an open filter get through.

More formally:

• Boolean indexing is the process of selecting rows (or elements) from an array or DataFrame based on a boolean condition.
• The condition is typically a logical expression like df['age'] > 30, which produces a boolean Series of the same length as the DataFrame.
• When you place that boolean Series inside square brackets (df[mask]), pandas keeps only the rows where the mask is True.

The same concept applies in NumPy with arrays: arr[arr > 5] returns the elements greater than 5.

Here's a simple analogy: imagine you have a list of all your emails. You create a checklist with True for emails you want to keep and False for spam. Boolean indexing is just handing the list and the checklist to Python, and it instantly returns only the True ones.

How it works step by step

Let's break down the process of filtering rows with boolean indexing into clear steps:

1. Start with a DataFrame or array — your data.
2. Write a condition that operates on one or more columns. For example, df['price'] > 100 returns a boolean Series.
3. Combine conditions if needed, using the bitwise operators & (AND), | (OR), ~ (NOT), and parentheses to group.
4. Apply the mask by placing it inside square brackets after the DataFrame: df[mask].
5. Optionally, assign the result to a new variable for further use.

It's important to understand that the mask must be the same length as the DataFrame (or array). If it's not, you'll get an error.

Hands-on walkthrough

Let's get our hands dirty with real code. We'll use pandas and NumPy to demonstrate filtering with boolean indexing.

Setting up the data

First, import the libraries and create a sample DataFrame:

import pandas as pd
import numpy as np

# Sample sales data
data = {
    'product': ['Laptop', 'Mouse', 'Keyboard', 'Monitor', 'USB-C Hub'],
    'price': [1200, 25, 80, 300, 45],
    'units_sold': [150, 900, 500, 200, 750]
}

df = pd.DataFrame(data)
print(df)

Output:

    product  price  units_sold
0    Laptop   1200         150
1     Mouse     25         900
2  Keyboard     80         500
3   Monitor    300         200
4  USB-C Hub     45         750

Basic boolean filtering

Now, let's filter rows where the price is greater than 100:

mask = df['price'] > 100
print(mask)

Output:

0     True
1    False
2    False
3     True
4    False
Name: price, dtype: bool

Now apply the mask:

expensive = df[mask]
print(expensive)

Output:

   product  price  units_sold
0   Laptop   1200         150
3  Monitor    300         200

Combining conditions with &, |, ~

Need more complex filters? Combine conditions using bitwise operators. Don't forget parentheses!

# Products that are cheap (< 100) and sold well (> 600 units)
filtered = df[(df['price'] < 100) & (df['units_sold'] > 600)]
print(filtered)

Output:

    product  price  units_sold
1     Mouse     25         900
4  USB-C Hub     45         750

# Products that are expensive (>= 1000) OR sold poorly (<= 200)
filtered2 = df[(df['price'] >= 1000) | (df['units_sold'] <= 200)]
print(filtered2)

Output:

  product  price  units_sold
0  Laptop   1200         150
3 Monitor    300         200

Using ~ to negate a condition

Need everything except what matches? Use the tilde operator:

not_mouse = df[~(df['product'] == 'Mouse')]
print(not_mouse)

Output:

    product  price  units_sold
0    Laptop   1200         150
2  Keyboard     80         500
3   Monitor    300         200
4  USB-C Hub     45         750

Working with NumPy arrays

The same principle works with NumPy:

import numpy as np

arr = np.array([10, 25, 3, 40, 5])
filtered_arr = arr[arr > 10]
print(filtered_arr)

Output:

[25 40]

Compare options / when to choose what

While boolean indexing is the go-to method, pandas offers other ways to filter rows. Here's a quick comparison:

When to use what? - Use boolean indexing for most value-based filters, especially when combining multiple conditions. - Use .query() when you have long conditions and want readability, or when you're coming from SQL. - Use .loc[] when you also need to select specific columns, or when you need to set values based on a condition. - Use .filter() only for label-based selection (e.g., columns starting with a string).

Troubleshooting & edge cases

Even experienced data scientists stumble on a few common issues. Here's how to fix them fast.

ValueError: The truth value of a Series is ambiguous

This happens when you try to use and, or, or not on a boolean Series instead of &, |, ~.

Wrong:

# This raises an error
df[(df['price'] < 100) and (df['units_sold'] > 600)]

Fix: Use & and parentheses:

df[(df['price'] < 100) & (df['units_sold'] > 600)]

Length mismatch errors

If your mask has a different length than the DataFrame, you'll get an error. This often happens when you forget to reset the index after a previous filter.

Wrong:

subset = df[df['price'] > 100]
# Now subset has only 2 rows, but df has 5
mask = subset['price'] < 500  # length 2
df[mask]  # ERROR!

Fix: Always apply the mask to the same object you created it from, or reset the index:

subset = df[df['price'] > 100].reset_index(drop=True)
mask = subset['price'] < 500
subset[mask]  # works fine

Missing values (NaN) in the condition column

If your data has NaN, comparisons like > or < return False for those rows, which may be unexpected.

import numpy as np
df_with_na = df.copy()
df_with_na.loc[2, 'price'] = np.nan
print(df_with_na[df_with_na['price'] > 50])

Output: It will exclude the row with NaN.

To include them, use isna():

mask = (df_with_na['price'] > 50) | df_with_na['price'].isna()

Using .loc for filtering vs. indexing

Sometimes you may accidentally use chained indexing, which can lead to unpredictable behavior. Prefer .loc when combining row and column selection:

# Instead of
df[df['price'] > 100]['product']

# Use
row_mask = df['price'] > 100
df.loc[row_mask, 'product']

Performance with large datasets

Boolean indexing is vectorized, so it's fast. But if you're doing many operations, consider using NumPy arrays or avoiding repeated copies. For very large frames, you may also want to use the numexpr engine in .query() for extra speed.

Pro tip: Use .copy() when you plan to modify a filtered dataframe, to avoid SettingWithCopyWarning.

What you learned & what's next

You've just mastered filter rows with boolean indexing — a core data-wrangling skill. You now understand how to create boolean masks, combine them with &, |, ~, and apply them to DataFrames and NumPy arrays. You also learned how to avoid common pitfalls like ambiguity errors and length mismatches.

This lesson covered: - The problem of manual row selection - The mental model of a mask/filter - Step-by-step application of boolean indexing - Hands-on examples with pandas and NumPy - Comparison with alternative methods like .query() and .loc[] - Troubleshooting edge cases

You're now ready to move to the next lesson in the track, where we'll build on this foundation to filter and transform columns — a natural next step that will let you combine row selection with column manipulation to shape your data exactly as you need.

Keep practicing: try filtering a dataset you have on hand with at least three different conditions. The more you use boolean indexing, the more intuitive it becomes.