How to Merge Multiple Data Sources in Python
A beginner-friendly helper that merges lists of dictionaries from multiple sources into one combined list using key filtering.
Python code
31 linesimport json
def merge_pipeline_data(*data_sources, keys=()):
"""Merge multiple data sources (list of dicts) into a single list of merged dicts.
Args:
*data_sources: One or more lists of dictionaries.
keys: Tuple of keys to include from each source (empty means all keys).
Returns:
Merged list of dictionaries.
"""
merged = []
for idx, source in enumerate(data_sources):
for record in source:
filtered_record = {k: v for k, v in record.items() if k in keys} if keys else record
if len(merged) <= idx:
merged.append({})
merged[idx].update(filtered_record)
return merged
if __name__ == "__main__":
users = [
{"id": 1, "name": "Alice", "email": "alice@example.com"},
{"id": 2, "name": "Bob", "email": "bob@example.com"}
]
orders = [
{"id": 1, "product": "Laptop", "cost": 1200},
{"id": 2, "product": "Keyboard", "cost": 75}
]
result = merge_pipeline_data(users, orders, keys=("id", "name", "product", "cost"))
print(json.dumps(result, indent=2))
Output
[
{
"id": 1,
"name": "Alice",
"product": "Laptop",
"cost": 1200
},
{
"id": 2,
"name": "Bob",
"product": "Keyboard",
"cost": 75
}
]
How it works
The merge_pipeline_data function iterates over each supplied data source and each record inside it. If a keys tuple is provided, only those keys are copied from the source dictionary; otherwise, all fields are kept. The positional index of the source determines the position of the merged dict in the final list, so sources are combined by row order. This pattern is common in data pipelines where related records from different tables must be joined by their index.
Common mistakes
- Assuming sources are merged by a shared key like 'id' instead of by index
- Not passing a keys tuple when you want only a subset of fields
- Mixing sources of different lengths will drop the extra rows from the longer source
Variations
- Use pandas.merge to join by a key column when you need explicit join semantics
- Use dict comprehension with zip to merge two lists by index more concisely
Real-world use cases
- Combining user and order CSV exports by row into a single analytics table
- Blending output from multiple API endpoints into one denormalized report
- Merging configuration dictionaries from separate environment files for a service
Sponsored
More from Data pipelines & processing
- Add a UUID Surrogate Key to Each Row in a CSV with Python easy
- Attach Source File Metadata to Records in Python easy
- Build a Python Utility That Detects Duplicate Records Across Multiple Excel Sheets medium
- Check Null Rate Threshold in PySpark DataFrame medium
- Count Records Processed per Category in Python easy
- Create Data Helper Functions in Python for Beginners easy
Keep learning
Related tutorials and quizzes for this topic.