Filter Data with Boolean Conditions

You've loaded your DataFrame, cleaned a few columns, maybe even grouped a bit — but now you're facing the real moment of truth: which rows actually matter? If you've ever written a loop just to pull out every customer over 30 or every sale above $500, you know the pain: verbose code, slow execution, and bugs hiding in plain sight. Filter data with boolean conditions is the skill that turns that struggle into a one-liner. It's the difference between wrestling with data and letting it speak to you.

The problem this lesson solves

Manually checking every row with a for loop feels natural at first — until your dataset has 100,000 rows, or 10 million. Loops are slow, hard to read, and easy to get wrong. Consider this familiar scene:

# Painful, slow, and error-prone
high_sales = []
for index, row in df.iterrows():
    if row['revenue'] > 500 and row['region'] == 'West':
        high_sales.append(row)

result = pd.DataFrame(high_sales)

It works, but it's fragile. One typo in a column name, one wrong type, and your analysis silently breaks. Worse, it doesn't scale. Filtering with boolean conditions replaces this procedural mess with vectorized, declarative, and lightning-fast operations. You'll stop writing how to select rows and start writing what you want — and Python's data stack will do the rest.

Core concept / mental model

Think of a DataFrame as a spreadsheet with labeled columns and rows. A boolean condition is a yes/no question you ask about each row: Is revenue greater than 500? The result of that question is a boolean mask — a Series of True and False values, one per row.

• True means the row meets your condition and gets kept.
• False means it's filtered out.

You then apply that mask to the DataFrame using .loc or square brackets, and the rows with True appear in your result. It's like holding a stencil over your data: only the parts where the stencil has holes (the True values) show through.

In pandas, this pattern is often called boolean indexing. The key components are:

• Comparison operators: >, <, >=, <=, ==, !=
• Logical operators: & (and), | (or), ~ (not) — note: you use and, or, not in plain Python, but pandas is different.
• The mask: a pandas Series of booleans, same length as your DataFrame

A mental model to lock in: You're not iterating; you're asking the data a question and letting it answer row by row.

How it works step by step

Let's walk through the mechanics of filtering with boolean conditions in pandas.

Step 1: Start with your DataFrame

You have a table of rows and columns. Each column has a dtype, and each row represents an observation.

import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Diana'],
    'age': [25, 32, 37, 29],
    'salary': [48000, 62000, 75000, 55000]
})

Step 2: Build a boolean condition

Pandas applies the comparison element-wise. df['age'] > 30 returns a Series of True/False.

mask = df['age'] > 30
print(mask)

Output:

0    False
1     True
2     True
3    False
Name: age, dtype: bool

Step 3: Apply the mask to the DataFrame

Use .loc[mask] or df[mask] — both work. .loc is explicit and recommended.

result = df.loc[mask]
print(result)

Output:

      name  age  salary
1      Bob   32   62000
2  Charlie   37   75000

Step 4: Combine conditions

Use & for AND, | for OR, and always wrap each condition in parentheses.

# Age over 30 AND salary above 60000
result = df.loc[(df['age'] > 30) & (df['salary'] > 60000)]

Hands-on walkthrough

Time to get your hands dirty. We'll use a realistic sales DataFrame and filter through several scenarios.

Setup: Create a sample dataset

import pandas as pd

df = pd.DataFrame({
    'order_id': [101, 102, 103, 104, 105],
    'customer': ['Ava', 'Liam', 'Emma', 'Noah', 'Olivia'],
    'region': ['East', 'West', 'West', 'East', 'South'],
    'revenue': [150, 220, 300, 175, 90],
    'units_sold': [2, 3, 5, 2, 1]
})

Example 1: Single condition

Filter rows where revenue is at least 200.

high_revenue = df.loc[df['revenue'] >= 200]
print(high_revenue)

Output:

   order_id customer region  revenue  units_sold
1       102     Liam   West      220           3
2       103     Emma   West      300           5

Example 2: Multiple conditions with AND and OR

Find orders from the West region with revenue above 200, or any order with units sold equal to 1.

# Condition 1: West and revenue > 200
# Condition 2: units_sold == 1
result = df.loc[((df['region'] == 'West') & (df['revenue'] > 200)) | (df['units_sold'] == 1)]
print(result)

Output:

   order_id customer region  revenue  units_sold
1       102     Liam   West      220           3
2       103     Emma   West      300           5
4       105   Olivia  South       90           1

Example 3: Invert a condition with ~

Get all rows where region is not West.

not_west = df.loc[~(df['region'] == 'West')]
print(not_west)

Output:

   order_id customer region  revenue  units_sold
0       101      Ava   East      150           2
3       104     Noah   East      175           2
4       105   Olivia  South       90           1

Example 4: Use .isin() for membership

Combine filtering with a list of allowed values — it reads cleaner than multiple ==.

result = df.loc[df['region'].isin(['East', 'South'])]

Compare options / when to choose what

Not all filtering is created equal. Here's how the main methods stack up:

When to use what: For simple conditions, df[mask] is fine. For anything more complex — or if you're chaining operations — switch to .loc or .query(). If you're coming from SQL, df.query('col1 > 0 & col2 <= 100') will feel instantly familiar.

Troubleshooting & edge cases

Even experienced pandas users stumble here. Here are the classic traps and how to escape them.

Problem: and, or, not don't work

You write df[(df['a'] > 1) and (df['b'] < 2)] and get ValueError: The truth value of a Series is ambiguous.

• Cause: pandas can't decide if the whole Series is true or false.
• Fix: Use bitwise operators &, |, ~ and wrap each condition in parentheses.

Problem: & vs and in NumPy arrays

The same rule applies to NumPy arrays. Always use &, |, ~.

Problem: Missing values in the condition column

If your column has NaN, comparisons like df['col'] > 10 return False for those rows — they get silently dropped. Decide if that's intended.

# Keep rows where col > 10 OR col is missing
result = df.loc[(df['col'] > 10) | df['col'].isna()]

Problem: isin() with duplicate values

Duplicates in the list don't matter — membership is boolean. No problem there.

Problem: Strings and case sensitivity

df['city'] == 'new york' won't match 'New York'. Normalize before comparing:

df.loc[df['city'].str.lower() == 'new york']

Pro tip: avoid chained indexing

Avoid doing df[df['a'] > 1]['b'] — you might get a SettingWithCopyWarning when you try to modify. Use .loc:

df.loc[df['a'] > 1, 'b'] = 0  # safe

What you learned & what's next

You now know how to filter data with boolean conditions in pandas: what a boolean mask is, how to apply it with .loc, how to combine conditions with & and |, and how to handle common edge cases. You've completed a practical exercise across four realistic filtering scenarios.

You've also seen how .isin() and .query() can make your code cleaner. These skills are the backbone of every data analysis — you'll use them constantly to focus on the data that actually answers your question.

Next in this track, you'll learn how to group and aggregate filtered data — turning these focused subsets into meaningful summaries like sums, means, and counts. That's where the real insight starts. But first, a quick recap: Filtering is fast, readable, and vectorized — ditch the loops and let pandas do the work.

Pro tip: When you write a filter, read it aloud: “Orders from the West with revenue above 200” maps directly to df.loc[(df['region'] == 'West') & (df['revenue'] > 200)]. If the sentence works, your code will too.