Select Columns and Filter Rows

You've loaded your dataset, cleaned it a bit, and now you're staring at a DataFrame with 50 columns and 100,000 rows. Where do you start? Scrolling through all that data is a productivity killer — you need to zoom in on just the columns you care about and the rows that matter. This lesson gives you the two most important pandas superpowers for that task: selecting columns and filtering rows. You'll learn the exact syntax, the mental model behind it, and the edge cases that trip up even experienced developers. By the end, you'll be slicing and dicing DataFrames with confidence, ready to move on to more advanced data manipulation.

The Problem This Lesson Solves

DataFrames are powerful, but their very richness creates a problem: too much information all at once. Imagine a DataFrame from a customer survey with columns like age, income, city, satisfaction, purchase_history, and 45 more. You need to analyze only age and satisfaction for customers over 30. Without the ability to select columns and filter rows, you would have to:

• Print the entire DataFrame and manually note the rows that match your condition — hopeless for millions of rows.
• Write slow, manual Python loops to extract data — error-prone and painfully slow.
• Waste memory and processing time on data you don't need.

The solution is concise, expressive, and blazing fast: pandas' built-in indexing and boolean filtering. Mastering these two operations is a prerequisite for every data science task that follows — grouping, merging, visualizing, modeling. In short, this lesson turns 'overwhelmed by data' into 'I see exactly what I need'.

Core Concept / Mental Model

Think of a pandas DataFrame as a table with labels — both columns and rows have names (the index). Selecting columns and filtering rows are two sides of the same coin: subsetting by labels vs. subsetting by conditions.

• Column selection is like choosing which 'files' to open from a drawer. You pick the columns you want by name or by positional number.
• Row filtering is like putting a 'filter' on a sieve — you keep only the rows that pass a boolean test (e.g., age > 30).

Under the hood, pandas treats row filtering as boolean masking: you create a Series of True/False values, one per row, and pandas keeps rows where the mask is True. This mask is then used with the .loc accessor to pick the exact rows and columns.

Pro tip: A boolean mask is a Series with the same index as the DataFrame. If the index isn't aligned, your filter will silently produce wrong results — always check your index!

Here's the key vocabulary you'll use:

• DataFrame: the 2D labeled table.
• Series: a single column (1D) — your boolean mask is a Series.
• Index: the row labels (often 0..n-1, but can be anything).
• Column labels: the names you use to refer to columns.

When you combine column selection and row filtering, you get a subset — a smaller DataFrame (or a Series) that contains only the data you need. That's the foundation of all exploratory data analysis.

How It Works Step by Step

Let's break down the process into logical steps, from basic to combined.

Step 1: Select columns with bracket notation

The simplest way to grab one column is df['column_name'] — this returns a Series. For multiple columns, pass a list of names: df[['col1', 'col2']] — this returns a DataFrame.

import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 32, 37],
    'city': ['NYC', 'LA', 'Chicago']
})

# One column -> Series
ages = df['age']
print(type(ages), ages)

# Multiple columns -> DataFrame
subset = df[['name', 'city']]
print(type(subset), subset)

Expected output:

<class 'pandas.core.series.Series'> 0    25
1    32
2    37
Name: age, dtype: int64
<class 'pandas.core.frame.DataFrame'>      name      city
0    Alice       NYC
1      Bob        LA
2  Charlie   Chicago

Step 2: Filter rows with a boolean condition

To keep only rows that meet a condition, write a comparison that produces a boolean Series, then use it to index the DataFrame: df[df['age'] > 30].

# Filter rows where age > 30
over_30 = df[df['age'] > 30]
print(over_30)

Expected output:

      name  age    city
1      Bob   32      LA
2  Charlie   37  Chicago

Step 3: Combine column selection and row filtering with .loc

The .loc accessor lets you specify both rows and columns by label in one go: .loc[rows, columns]. The rows part can be a boolean mask, and the columns part can be a list of column names.

# Select 'name' and 'age' for rows where age > 30
result = df.loc[df['age'] > 30, ['name', 'age']]
print(result)

Expected output:

      name  age
1      Bob   32
2  Charlie   37

Step 4: Add more complex conditions

Combine multiple conditions with &, |, and ~ (remember to wrap each condition in parentheses). Use .isin() to filter on a list of values.

# Age > 25 AND city == 'Chicago'
filtered = df[(df['age'] > 25) & (df['city'] == 'Chicago')]
print(filtered)

# Age 30-35 OR city in ['NYC', 'LA']
filtered2 = df[(df['age'].between(30, 35)) | (df['city'].isin(['NYC', 'LA']))]
print(filtered2)

Expected output:

      name  age    city
2  Charlie   37  Chicago

      name  age    city
0    Alice   25     NYC
1      Bob   32      LA
2  Charlie   37  Chicago

Hands-on Walkthrough

Let's apply all this to a realistic dataset: a sales records DataFrame. We'll practice the full workflow — load, inspect, then subset.

Setup: Create sample data

import pandas as pd

sales = pd.DataFrame({
    'order_id': [101, 102, 103, 104, 105],
    'product': ['Laptop', 'Mouse', 'Keyboard', 'Monitor', 'Laptop'],
    'price': [1200, 25, 80, 300, 1250],
    'quantity': [1, 3, 2, 1, 1],
    'city': ['NYC', 'LA', 'Chicago', 'NYC', 'LA']
})

# See the structure
print(sales.head())

Expected output:

   order_id   product  price  quantity    city
0       101    Laptop   1200         1     NYC
1       102     Mouse     25         3      LA
2       103  Keyboard     80         2  Chicago
3       104   Monitor    300         1     NYC
4       105    Laptop   1250         1      LA

Exercise 1: Select columns

Your goal: extract only product and price for analysis.

# Your code here
selected = sales[['product', 'price']]
print(selected)

Expected output:

    product  price
0    Laptop   1200
1     Mouse     25
2  Keyboard     80
3   Monitor    300
4    Laptop   1250

Exercise 2: Filter rows

Your goal: find all orders with a price greater than $100.

# Your code here
expensive = sales[sales['price'] > 100]
print(expensive)

Expected output:

   order_id   product  price  quantity    city
0       101    Laptop   1200         1     NYC
3       104   Monitor    300         1     NYC
4       105    Laptop   1250         1      LA

Exercise 3: Combine both

Your goal: show product and quantity for all orders in 'NYC' or with quantity > 1.

# Your code here
subset = sales.loc[(sales['city'] == 'NYC') | (sales['quantity'] > 1), ['product', 'quantity']]
print(subset)

Expected output:

    product  quantity
0    Laptop         1
1     Mouse         3
2  Keyboard         2
3   Monitor         1

Pro tip: Always assign the result of a filter to a new variable (like filtered or subset). If you don't, you'll keep re-applying the condition to the original DataFrame and wonder why changes don't stick.

Compare Options / When to Choose What

When selecting columns and filtering rows, you have several tools. Each has its place.

💡 Which should you use? - For a quick single column, use df['col']. - For boolean row filtering, df[mask] is simplest. - For combining both with clarity, use .loc[rows, columns]. - If you need positional slicing (e.g., first 3 rows, columns 2–4), use .iloc. - For column selection based on naming patterns, filter is handy.

When to avoid? - Don't use .iloc with column names — it expects integers. - Don't use chained assignment like df[df['a']>5]['col'] = 0 — it may warn and fail; use .loc instead.

Troubleshooting & Edge Cases

Boolean operators and/or fail on Series

You might think you can write df[(df['age'] > 25) and (df['age'] < 35)], but Python throws ValueError: The truth value of a Series is ambiguous. Use & and | instead, and wrap each condition in parentheses.

Negative filtering with ~

To select rows that do NOT match a condition, use ~ before the mask: df[~df['city'].isin(['NYC', 'LA'])]]. Forgetting the tilde gives you the opposite result.

Missing values and NaN

If a column contains NaN, comparisons like df['col'] > 5 will exclude those rows (the comparison returns False for NaN). To keep them, combine with df['col'].isna().

Index alignment issues

If your mask's index doesn't match the DataFrame's index, you'll get NaN or empty results. Always ensure the mask comes from the same DataFrame.

SettingWithCopyWarning

When you filter with df[mask] and then try to modify the result, pandas may warn you because you might be modifying a copy. Use .loc to be safe:

# Safer way to set values on filtered rows
df.loc[df['price'] > 100, 'discount'] = 0.1

What You Learned & What's Next

Great job! You've mastered the core superpowers of pandas:

• You can select columns using bracket notation and lists.
• You can filter rows using boolean conditions with df[mask].
• You can combine both elegantly with .loc[rows, columns].
• You know how to use &, |, ~, .isin(), and .between() for complex filters.
• You're aware of common pitfalls like the and/or error and SettingWithCopyWarning.

Now that you can grab any subset of your data, you're ready for the next lesson in this track: grouping and aggregating data — the next logical step to turn subsets into insights. You'll learn to answer questions like 'What is the average price per city?' using groupby(). Keep practicing, and you'll be manipulating DataFrames like a pro in no time!