Create New Columns

Ever stared at a raw dataset and thought, “I wish this column summed those two, or told me if the price changed, or flagged this row as urgent”? Raw data almost never arrives in the shape you need. You’ll spend a large part of your data science life creating new columns from existing data — combining, transforming, and deriving fields that unlock the real signal. This lesson gives you the pandas toolbox to do that cleanly, efficiently, and without the silent bugs that quietly corrupt your analysis.

The problem this lesson solves

When you load a CSV or pull data from an API, you get exactly what the source gave you — not what your analysis needs. Maybe you need:

• A total_price from quantity * unit_price
• A status column that says "high", "medium", or "low" based on a numeric score
• A date_diff between an order date and a ship date

Writing this logic outside your DataFrame means copying values around, losing alignment, and creating errors the moment rows shift or get filtered. The core problem: you need to derive new fields inside your dataset, reliably, and with code that’s obvious to your future self (and your teammates).

This lesson focuses on pandas — the de facto tool for tabular data in Python. By the end, you’ll confidently turn existing columns into exactly the features your analysis or model needs.

Core concept / mental model

Think of your DataFrame as a spreadsheet on steroids. Each column is a series of values, all aligned by row index. Creating a new column is like adding a new column to that spreadsheet — but here, you write formulas (Python expressions) instead of clicking and dragging.

The mental model breaks down into three operations:

1. Derive — compute a new value from one or more existing columns (e.g., df['area'] = df['width'] * df['height'])
2. Transform — map or convert existing values (e.g., df['lower_name'] = df['name'].str.lower())
3. Categorize — assign labels or flags based on conditions (e.g., df['is_adult'] = df['age'] >= 18)

Importantly, pandas aligns on the index — so even messier data sources line up correctly. You can also use df.assign() for a functional style, which returns a new DataFrame rather than modifying your original — a habit that prevents accidental data corruption.

Pro tip: Always ask: Do I want to modify in place (df['col'] = ...) or return a new object (df.assign(...))? In a pipeline, prefer assign for a clean, functional flow.

How it works step by step

The mechanics of adding a column are straightforward, but the choices matter. Here’s the logical sequence:

1. Load or have your DataFrame ready — usually from a CSV, database, or API.
2. Identify the existing columns you’ll use as inputs.
3. Write your expression — a Python operation, a string method, a NumPy function, or a lambda.
4. Assign the result to a new column name — either with df['new'] = ... or df.assign(new=...).
5. Check the result — print(df.head()) or df.info() to confirm the new column exists and looks right.

Let’s see each step in action.

Hands-on walkthrough

We’ll use a small sales dataset to practice. First, create a DataFrame from a list of dicts — this is a realistic starting point.

import pandas as pd

sales = pd.DataFrame({
    'product': ['Laptop', 'Keyboard', 'Mouse', 'Monitor'],
    'quantity': [2, 5, 10, 1],
    'unit_price': [999.0, 49.0, 25.0, 249.0],
    'order_date': ['2024-01-15', '2024-01-16', '2024-01-17', '2024-01-18']
})

print(sales)

Output:

  product  quantity  unit_price   order_date
0  Laptop         2       999.0  2024-01-15
1  Keyboard       5        49.0  2024-01-16
2  Mouse         10        25.0  2024-01-17
3  Monitor        1       249.0  2024-01-18

Now, create a new column total_price by multiplying two existing columns:

# Option A: direct assignment (modifies the original DataFrame)
sales['total_price'] = sales['quantity'] * sales['unit_price']
sales['is_expensive'] = sales['total_price'] > 500

print(sales)

Output:

  product  quantity  unit_price   order_date  total_price  is_expensive
0  Laptop         2       999.0  2024-01-15       1998.0          True
1  Keyboard       5        49.0  2024-01-16        245.0         False
2  Mouse         10        25.0  2024-01-17        250.0         False
3  Monitor        1       249.0  2024-01-18        249.0         False

But what about more complex logic? You can use apply with a lambda or a custom function to compute values that don’t fit a simple arithmetic expression.

# Determine a shipping label based on total price
def shipping_label(price):
    if price >= 1000:
        return 'Free Express'
    elif price >= 200:
        return 'Free Standard'
    else:
        return 'Standard'

sales['shipping'] = sales['total_price'].apply(shipping_label)

print(sales[['product', 'total_price', 'shipping']])

Output:

  product  total_price     shipping
0  Laptop       1998.0  Free Express
1  Keyboard      245.0  Free Standard
2  Mouse         250.0  Free Standard
3  Monitor       249.0  Free Standard

Adding date-based columns is just as easy. Convert the date strings to datetime, then add a day_of_week column.

sales['order_dt'] = pd.to_datetime(sales['order_date'])
sales['day_of_week'] = sales['order_dt'].dt.day_name()

print(sales[['product', 'order_date', 'day_of_week']])

Output:

  product   order_date day_of_week
0  Laptop  2024-01-15       Monday
1  Keyboard 2024-01-16      Tuesday
2  Mouse   2024-01-17    Wednesday
3  Monitor 2024-01-18    Thursday

Pro tip: Use df.assign() in a method chain to keep your code tidy:

sales = sales.assign(
    total_price=sales['quantity'] * sales['unit_price'],
    is_expensive=sales['quantity'] * sales['unit_price'] > 500,
    shipping=sales['quantity'] * sales['unit_price'].apply(shipping_label)
)

Compare options / when to choose what

You have several ways to add a column — each fits a different scenario. Here’s the decision table:

When to choose what:

• Vectorized arithmetic (e.g., df['a'] * 2) — fastest, always preferred when possible.
• assign — when you’re building a multi-step transformation pipeline.
• apply with axis=1 — for complex row-based calculations (but be aware it’s slower for large DataFrames).
• np.where — for simple conditional flags without writing a function.

Troubleshooting & edge cases

Creating columns goes wrong in predictable ways. Here’s how to fix them:

1. KeyError: 'column_name'

You’re referencing a column that doesn’t exist — usually a typo or you forgot to inspect your data.

# Wrong: 'quantty' is misspelled
df['total'] = df['quantty'] * df['unit_price']

Fix: Print df.columns and check exact names before referencing.

2. ValueError: operands could not be broadcast together

You’re trying to combine Series of different lengths — often because you filtered one column without resetting the index.

# Wrong: y is filtered, x is not
y = df[df['qty'] > 0]
df['new'] = df['price'] * y['qty']

Fix: Use reset_index(drop=True) on the filtered Serie or avoid filtering separate columns entirely.

3. Silently wrong values (the worst one)

If you use a Python list directly instead of a Series, pandas aligns by position, not by index. If your index is not 0,1,2..., the new column will be misaligned.

# Wrong: uses a list, index gaps cause misalignment
df = df[df['qty'] > 0]  # index now 0 and 2
df['total'] = [100, 200]  # intends qty*unit_price, but places 100 and 200 at rows 0 and 1

Fix: Always compute from Series (e.g., df['qty'] * df['unit_price']) or use reset_index(drop=True) after filtering.

4. Date columns remain strings

If you try sales['order_dt'].dt.day_name() but order_dt is a string, you’ll get an AttributeError.

Fix: Convert with pd.to_datetime() first.

What you learned & what's next

You can now create new columns from existing data — the bread-and-butter of data preparation. You’ve learned:

• How to derive numeric columns with arithmetic (covered learning objective 1: explain the core idea)
• How to use apply for custom logic and assign for clean pipelines
• How to handle date-derived columns and simple flags
• How to complete a practical exercise in the hands-on section (learning objective 2)
• Real-world troubleshooting: alignment, KeyError, and dtype pitfalls

What’s next: In the next lesson, you’ll learn how to filter and query your data — selecting the exact rows you need for analysis. You now have the tools to shape datasets, and filtering is the next step toward clean, focused datasets.