Union Multiple DataFrames with Aligned Columns in Python

Concatenate DataFrames with different columns, aligning them and filling missing values with NaN using pandas concat.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 14 views 0 copies

pandas dataframes concat data-wrangling schemas

Requires third-party packages — install first

pip install pandas

Python code

32 lines

Python 3.9+

import pandas as pd
from io import StringIO

# Sample dataframes with different columns
df1 = pd.DataFrame({
    'id': [1, 2, 3],
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35]
})

df2 = pd.DataFrame({
    'id': [4, 5],
    'name': ['Diana', 'Eve'],
    'city': ['NYC', 'LA']
})

df3 = pd.DataFrame({
    'id': [6],
    'age': [40],
    'score': [88.5]
})

# Union multiple schemas: align columns, fill missing with NaN
def union_schemas(*dfs):
    return pd.concat(dfs, axis=0, join='outer', sort=False, ignore_index=True)

result = union_schemas(df1, df2, df3)

print("Combined DataFrame (aligned columns, missing filled with NaN):")
print(result.to_string())
print("\nColumn dtypes after alignment:")
print(result.dtypes.to_string())

Output

stdout

Combined DataFrame (aligned columns, missing filled with NaN):
   id      name   age   city  score
0   1     Alice  25.0    NaN    NaN
1   2       Bob  30.0    NaN    NaN
2   3   Charlie  35.0    NaN    NaN
3   4     Diana   NaN    NYC    NaN
4   5       Eve   NaN     LA    NaN
5   6       NaN  40.0    NaN   88.5

Column dtypes after alignment:
id         int64
name      object
age      float64
city      object
score     float64
dtype: object

How it works

The pd.concat function with axis=0 stacks DataFrames vertically. Using join='outer' ensures all columns from every input are preserved, so missing values appear as NaN. ignore_index=True resets the index to a continuous range, making the combined frame easy to work with. The resulting dtypes show that age becomes float64 due to NaN representation, which is expected behavior in pandas.

Common mistakes

Using `join='inner'` which drops columns not present in all frames
Forgetting `ignore_index=True` causing duplicate index values
Not sorting columns with `sort=False` which may reorder them unexpectedly

Variations

Use `df1.append([df2, df3], ignore_index=True)` for an alternative syntax (deprecated in pandas 2.0)
Instead of `pd.concat`, you can explicitly align with `df1.align(df2, join='outer')` but it's less efficient for many frames

Real-world use cases

Merging daily export files from different sources where each file has different columns.
Combining historical data snapshots from various versions of the same schema.
Building a unified dataset from API responses that return inconsistent fields.

Union Multiple DataFrames with Aligned Columns in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Keep learning

Tutorials

Quizzes