Series & DataFrame Basics

You’ve cleaned your arrays with NumPy and maybe even loaded a CSV, but now you stare at a real dataset — thousands of rows, mixed types, missing values — and every manual loop or list comprehension feels brittle. The pain is real: how do you slice, filter, group, and transform tabular data without writing 200 lines of error-prone Python? This lesson introduces pandas, the library that turns raw tabular data into two intuitive structures — Series (a labeled column) and DataFrame (a labeled table) — and gives you the exact mental tools to start analyzing data like a pro.

The problem this lesson solves

Without pandas, handling tabular data in pure Python is painful. You juggle lists of lists, track column indexes manually, and write nested loops just to compute a mean or filter rows. Worse, a single missing value or type mismatch can silently corrupt your results or crash your script with a confusing TypeError.

Consider a simple scenario: you have sales data with columns date, product, revenue, and units_sold. To find total revenue per product, you'd need to write something like:

# Pure Python: verbose, fragile, and slow
sales = [
    ['2024-01-01', 'Widget', 120.50, 3],
    ['2024-01-01', 'Gadget', 99.99, 1],
    ['2024-01-02', 'Widget', 250.00, 5],
]

product_totals = {}
for row in sales:
    product = row[1]
    revenue = row[2]
    if product not in product_totals:
        product_totals[product] = 0
    product_totals[product] += revenue

print(product_totals)  # {'Widget': 370.5, 'Gadget': 99.99}

This works, but it breaks the moment you add a column, change order, or need to filter by date. You are essentially reinventing a database every time. The lesson's solution: pandas gives you a standardized, high-performance way to store, index, and manipulate data, so you can focus on the insight, not the plumbing.

Core concept / mental model

Think of a DataFrame as a spreadsheet in memory — it has rows and columns, column headers, and a row index. Each column is a Series, which is like a single column from that spreadsheet, with its own index and a uniform data type. A Series is essentially a NumPy array with labels; a DataFrame is a collection of Series that share the same index.

Mental model: If a DataFrame is a smart table, then a Series is a smart column. Both know their labels, can handle missing data, and support vectorized operations — meaning you can apply calculations to entire columns without looping.

Let's see them side by side:

• Series: one-dimensional, labeled array. Index maps labels to values.
• DataFrame: two-dimensional, potentially heterogeneous tabular structure. Each column can have a different type.

Here's a visual in words:

DataFrame (sales)
+---------+---------+---------+-------------+
| index   | date    | product | revenue     |
+---------+---------+---------+-------------+
| 0       | 2024-.. | Widget  | 120.50      |
| 1       | 2024-.. | Gadget  | 99.99       |
+---------+---------+---------+-------------+
   ^         ^         ^         ^
   |         |         |         +---- Series (column revenue)
   |         |         +-------------- Series (column product)
   +------------------------------ Index (row labels)

Both structures rely on an index — a sequence of labels (often integers, but can be dates or strings) that give context to each value. Without labels, you'd be back to guessing positions; with them, operations like df[df['revenue'] > 100] become expressive and safe.

Key insight: pandas is built on top of NumPy. When you create a Series, you are effectively wrapping a NumPy array with an index. This means performance stays high, and you get all of NumPy's vectorized goodness for free.

How it works step by step

Let's walk through the logical flow of creating and using Series and DataFrames.

1. Import pandas — the standard alias is import pandas as pd. You'll use this in virtually every analysis script.
2. Create a Series from a list, dictionary, or NumPy array. The index is inferred or you supply it explicitly.
3. Create a DataFrame from a dictionary of list-like objects, a list of dictionaries, or from a file (CSV, Excel, etc.). Each dictionary key becomes a column.
4. Inspect the structure — use .shape, .columns, .index, and .dtypes to verify what you have before diving into analysis.
5. Select data — use label-based selection (.loc) or position-based selection (.iloc). Column selection uses simple indexing (df['column']).
6. Apply operations — compute summary statistics (.mean(), .sum()), filter rows with boolean masks, add new columns, and handle missing values with .dropna() or .fillna().
7. Iterate and reason — always check the result after each step; pandas operations often return new objects, so make sure you assign them if you want to keep the changes.

Hands-on walkthrough

Time to open your interactive Python environment and follow along. We'll start with a Series, then build a DataFrame and do real data work.

Step 1: Create a Series

import pandas as pd

# From a list, with an explicit index
temperatures = pd.Series([22.5, 25.0, 19.8], index=['Mon', 'Tue', 'Wed'])
print(temperatures)

Output:

Mon    22.5
Tue    25.0
Wed    19.8
dtype: float64

Notice the index labels on the left — this is what makes a Series more than a plain NumPy array. You can access values by label, just like a dictionary.

print(temperatures['Tue'])      # 25.0
print(temperatures.mean())      # 22.433...
print(temperatures > 20)        # boolean Series for filtering

Step 2: Create a DataFrame

Now build a DataFrame from a dictionary. This is the most common way to create one from scratch.

import pandas as pd

data = {
    'product': ['Widget', 'Gadget', 'Widget', 'Gizmo'],
    'region': ['North', 'South', 'South', 'North'],
    'revenue': [120.50, 99.99, 250.00, 310.20],
    'units_sold': [3, 1, 5, 2]
}

sales = pd.DataFrame(data)
print(sales)

Output:

   product  region  revenue  units_sold
0   Widget   North   120.50           3
1   Gadget   South    99.99           1
2   Widget   South   250.00           5
3    Gizmo   North   310.20           2

Your DataFrame has an automatically generated integer index (0–3) and named columns. Now inspect it:

print(sales.shape)        # (4, 4)
print(sales.columns)      # Index(['product', 'region', 'revenue', 'units_sold'], dtype='object')
print(sales.dtypes)       # see column types
print(sales.describe())   # summary statistics for numeric columns

Step 3: Real operations

Let's select and transform data.

# Select a single column → Series
revenue_series = sales['revenue']
print(type(revenue_series))  # <class 'pandas.core.series.Series'>

# Filter rows: sales where revenue > 100
high_sales = sales[sales['revenue'] > 100]
print(high_sales)

Output:

   product  region  revenue  units_sold
0   Widget   North   120.50           3
2   Widget   South   250.00           5
3    Gizmo   North   310.20           2

Now let's add a calculated column and group by product:

# Add a new column: average price per unit
sales['avg_price'] = sales['revenue'] / sales['units_sold']
print(sales)

# Group by product and sum revenue
product_totals = sales.groupby('product')['revenue'].sum()
print(product_totals)

Output:

   product  region  revenue  units_sold  avg_price
0   Widget   North   120.50           3  40.166667
1   Gadget   South    99.99           1  99.990000
2   Widget   South   250.00           5  50.000000
3    Gizmo   North   310.20           2  155.100000

product
Gadget     99.99
Gizmo     310.20
Widget    370.50
Name: revenue, dtype: float64

See how clean that is? The same logic that took a manual loop now reads like a sentence. You've just used label-based indexing, boolean filtering, column creation, and vectorized division — all core skills for every future lesson.

Compare options / when to choose what

You might wonder: when should I use a Series versus a DataFrame? And when should I reach for pandas at all versus raw Python or NumPy? Here's a quick comparison.

Pro tip: Use pandas for anything beyond a handful of values. The learning curve is worth it — you'll save hours on every project.

For selection specifically, you'll often choose between .loc and .iloc:

• .loc[label] — select by index label (string or int from your index)
• .iloc[position] — select by integer position (like indexing a list)

print(sales.loc[2, 'revenue'])   # 250.0  (label row 2)
print(sales.iloc[2, 2])           # 250.0  (third row, third column)

If your index is default integers, both may give the same result, but the difference matters when your index has custom labels like dates or product names.

Troubleshooting & edge cases

Beginners hit a few predictable walls. Let's fix them before you fall in.

1. KeyError when selecting a column

Error: KeyError: 'revenue ' (note the trailing space)

Cause: Column name typos or whitespace from imported data.

Fix: Print sales.columns first. Trim whitespace with sales.columns = sales.columns.str.strip() if needed.

2. SettingWithCopyWarning when modifying a slice

# This can trigger a warning
high_sales = sales[sales['revenue'] > 100]
high_sales['bonus'] = 0  # Warning!

Cause: You're modifying a copy, not the original DataFrame.

Fix: Use .copy() to make your intent explicit: high_sales = sales[sales['revenue'] > 100].copy(). This avoids unpredictable side effects.

3. Mismatched indices when adding columns

new_prices = pd.Series([10, 20, 30], index=[0, 1, 2])
sales['price'] = new_prices  # leaves NaN for row 3

Cause: Your Series index doesn't match DataFrame index.

Fix: Align indices — make sure your Series has the same index as the DataFrame, or use .reset_index() on the Series to keep things tidy.

4. Missing values silently affect calculations

sales['revenue'].mean() returns NaN if any value is missing. This is by design, but it can surprise you.

Fix: Use df.dropna() to remove rows with missing values, or df.fillna(0) to replace them. Check df.isna().sum() to see where the holes are.

What you learned & what's next

You now have a working mental model of Series and DataFrame — pandas' two core data structures. You can create them, inspect them, select data, filter, add columns, and group by values. You've also learned to debug common pitfalls like index alignment and copy warnings. These skills are the foundation for everything else in data analysis: cleaning, merging, reshaping, and visualizing — all of which pandas makes dramatically easier.

Next lesson: Now that you can wrangle a DataFrame, it's time to load real data from files (CSV, Excel) and practice data cleaning — handling missing values, duplicates, and inconsistent types. This is where your Series/DataFrame skills become truly valuable on messy real-world datasets.