Combine Datasets with Concat and Append

You've cleaned your DataFrames, filtered out the noise, and computed group stats — but now the real world hits. Sales data arrives split by region. User logs come in daily chunks. Model features live in separate tables. If you've ever found yourself writing painful loops to glue DataFrames together, you know the pain: inconsistent schemas, duplicate rows, and that sinking feeling when your analysis silently drops half the data. This lesson ends that struggle. By learning to combine datasets with concat and append in pandas, you'll merge data like a pro — quickly, safely, and with full control over the outcome. Whether you're a beginner stitching your first datasets together or a developer leveling up, this is the skill that turns scattered data into analyzable insight.

The problem this lesson solves

Imagine you work for an e-commerce company. Sales for January, February, and March were exported as three separate CSV files because each month's data was too large for a single export. To analyze quarterly trends, you need one unified DataFrame with all 90 days of transactions. Copy-pasting rows into a spreadsheet? Not scalable. Writing a loop that reads each file and appends to a list? That's clunky, error-prone, and slow.

The core problem: when you have multiple DataFrames with the same structure — same columns, same data types, same purpose — you need a simple, reliable way to stack them vertically into one big DataFrame. Without this, you risk:

• Data loss: manually combining rows and accidentally skipping some.
• Inconsistency: different column orders or names causing mismatched data.
• Performance issues: loops and iterative appends that crawl on large datasets.

The solution is pandas' concat() function (and its older, now-deprecated sibling append()). These tools are built exactly for this job: taking multiple DataFrames and combining them into a single, analysis-ready structure.

Core concept / mental model

Think of your datasets as building blocks. Each DataFrame is a block, and concat() is the crane that stacks them together — either vertically (one on top of the other, default) or horizontally (side by side). The key insight is that concatenation is about stacking along an axis, not merging based on a key.

• Vertical stacking (axis=0): Aligns columns across DataFrames. If columns match, rows stack cleanly. If some columns are missing, pandas fills with NaN (not a number) — a subtle but crucial behavior.
• Horizontal stacking (axis=1): Aligns rows by index. This is like placing two tables side-by-side, but it's risky if indexes don't match up — you can get misaligned data or NaN-filled rows. Use with caution.

append() is a shortcut that internally calls concat(). It's been deprecated since pandas 2.0, so while you'll see it in older code, you should always prefer concat() for future-proof scripts. Think of append() as the older, less flexible cousin that adds one DataFrame to another; concat() is the modern, more powerful tool that handles multiple DataFrames at once.

A mental diagram:

Vertical (axis=0):
   [DF1]  →  [ DF1 ]
   [DF2]  →  [ DF2 ]
   [DF3]  →  [ DF3 ]

Horizontal (axis=1):
   [DF1] [DF2]  →  [ DF1 | DF2 ]

How it works step by step

Combining datasets with concat() involves a simple, repeatable process. Let's break it down:

1. Prepare your DataFrames: Ensure each DataFrame has the columns you need, ideally in the same order. If not, concat() will still work, but the result won't be as tidy.
2. Decide on the axis: For row-wise stacking (most common), use axis=0. For column-wise stacking, use axis=1.
3. Call concat(): Pass a list of DataFrames: pd.concat([df1, df2, df3]). You can also pass a dictionary to label each input, which creates a MultiIndex — useful for tracking origin.
4. Handle the index: By default, concat() preserves original indexes, which can lead to duplicate index values. Use ignore_index=True to get a fresh integer index, which is often cleaner for analysis.
5. (Optional) Adjust for missing columns: If your DataFrames have different columns, decide whether you want NaN for missing values (default) or to keep only common columns via join='inner'.

For append(): df1.append(df2) does the same as pd.concat([df1, df2]) with ignore_index=False. Remember, append() is deprecated — avoid in new code.

Cause → effect: When you stack vertically, pandas compares column names (not positions). If columns match, rows sit neatly. If columns mismatch, pandas fills gaps with NaN. This is why consistent column names are critical.

Hands-on walkthrough

Let's get practical. Fire up a Jupyter Notebook or Python script and follow along. We'll start with a classic scenario: monthly sales data.

Example 1: Vertical concatenation (stacking rows)

import pandas as pd

# Sales data for three months
sales_jan = pd.DataFrame({
    'date': ['2025-01-01', '2025-01-02'],
    'revenue': [100, 150]
})
sales_feb = pd.DataFrame({
    'date': ['2025-02-01', '2025-02-02'],
    'revenue': [200, 250]
})

# Combine into a single quarterly DataFrame
all_sales = pd.concat([sales_jan, sales_feb], ignore_index=True)
print(all_sales)

Output:

         date  revenue
0  2025-01-01      100
1  2025-01-02      150
2  2025-02-01      200
3  2025-02-02      250

Notice how index is reset to 0, 1, 2, 3 because we used ignore_index=True. This is usually what you want for analysis.

Example 2: Handling missing columns

Real-world data is messy. Maybe February has an extra column, or January lacks one. Let's see that behavior:

import pandas as pd

sales_jan = pd.DataFrame({'product': ['A', 'B'], 'revenue': [100, 150]})
sales_feb = pd.DataFrame({'product': ['C', 'D'], 'revenue': [200, 250], 'tax': [20, 25]})

combined = pd.concat([sales_jan, sales_feb], ignore_index=True)
print(combined)

Output:

  product  revenue   tax
0       A      100   NaN
1       B      150   NaN
2       C      200  20.0
3       D      250  25.0

January rows get NaN in the tax column. Interpretation is up to you — maybe tax didn't apply, maybe it was lost. Pro tip: Use pd.concat([...], join='inner') to drop columns that aren't present in all DataFrames if you only care about common columns.

Example 3: Horizontal concatenation (side-by-side)

Sometimes you have separate tables with the same index but different features. For instance, user demographics and purchase history:

import pandas as pd

users = pd.DataFrame({'user_id': [1, 2, 3], 'age': [25, 30, 35]}, index=[0,1,2])
purchases = pd.DataFrame({'total_spent': [100, 200, 150]}, index=[0,1,2])

# Combine horizontally, aligning on index
user_data = pd.concat([users, purchases], axis=1)
print(user_data)

Output:

   user_id  age  total_spent
0        1   25          100
1        2   30          200
2        3   35          150

Works great when indexes align. But if indexes don't match, you'll get NaN for missing rows — the classic pitfall we'll cover in troubleshooting.

Example 4: Using append (for legacy code)

Even though append is deprecated, you'll encounter it in old code. Here's how it works:

# Deprecated: avoid in new code
df = sales_jan.append(sales_feb, ignore_index=True)

Use concat() instead — it's faster, more flexible, and future-proof.

Compare options / when to choose what

Now, let's compare the main methods for combining datasets. This decision table will guide you:

When to choose what:

• Use concat() for vertical stacking when your DataFrames have the same columns but different rows — the bread and butter of this lesson.
• Use merge() when you need to join on a key (e.g., user_id) to bring in related information — that's a different lesson in this track.
• Use join() when your index is the key.

Variations: You can also pass a dictionary to concat() to label each input, creating a MultiIndex for grouping — handy for keeping track of data sources.

Troubleshooting & edge cases

Even with concat(), things can go sideways. Here are common issues and how to fix them:

Duplicate index values

If you don't use ignore_index=True, your result will have duplicated indexes, which can break later operations like loc[].

# Wrong: duplicates index
bad = pd.concat([df1, df2])
# Fix:
good = pd.concat([df1, df2], ignore_index=True)

Pro tip: Always use ignore_index=True unless you have a reason to keep original indexes.

Dtype surprises after concatenation

When columns are missing, pandas may change the dtype of a column to float64 to accommodate NaN. This often surprises beginners:

revenue_column_dtype = combined['revenue'].dtype  # float64 instead of int

Fix: Convert back to int after filling or dropping NaN values, or handle missing data intentionally.

Index alignment in horizontal concat

If indexes don't align when using axis=1, you get NaN rows. This is a silent killer — check your indexes!

# Wrong: mismatched indexes
users = pd.DataFrame({'age': [25, 30]}, index=[0,1])
purchases = pd.DataFrame({'total': [100]}, index=[1])
combined = pd.concat([users, purchases], axis=1)
print(combined)
# Output: age  total
# 0     25.0  NaN
# 1     30.0  100.0

Fix: Use reset_index() or set an appropriate index before concatenating, or use merge() if you need key-based alignment.

Performance with many small DataFrames

Using append() in a loop is notoriously slow. Instead, collect DataFrames in a list and call concat() once — this is significantly faster.

append() deprecation warning

If you see a FutureWarning about append, update your code to use concat(). It's not just cosmetic — append may be removed in future pandas versions.

What you learned & what's next

You now have a solid grasp of how to combine datasets with concat() and the deprecated append(). Specifically, you learned:

• The core concept of stacking DataFrames along rows (axis=0) or columns (axis=1), and when each is appropriate.
• The step-by-step process: prepare your DataFrames, choose the axis, call concat(), and handle the index.
• Practical examples covering vertical stacking, missing columns, and horizontal concatenation.
• How to choose between concat(), append(), merge(), and join() based on your use case.
• How to troubleshoot common pitfalls like duplicate indexes, dtype changes, and index misalignment.

Next step: You're ready to tackle merging datasets on a key — a more advanced combination technique that unlocks relational data analysis. That's the natural continuation of your data wrangling journey. Keep practicing — combine different datasets you encounter in your own work, and soon it'll be second nature.

Pro tip: Always inspect your concatenated result's shape and columns before proceeding with analysis. print(df.shape) is your best friend — catch problems early!

Now go ahead, grab two of your own DataFrames, and combine them. Then move to the next lesson to master merging on keys — you're on your way to becoming a data wrangling pro!