How to Merge Multiple Data Sources in Python

A beginner-friendly helper that merges lists of dictionaries from multiple sources into one combined list using key filtering.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 14 views 0 copies

Python code

31 lines
Python 3.9+
import json

def merge_pipeline_data(*data_sources, keys=()):
    """Merge multiple data sources (list of dicts) into a single list of merged dicts.
    
    Args:
        *data_sources: One or more lists of dictionaries.
        keys: Tuple of keys to include from each source (empty means all keys).
    Returns:
        Merged list of dictionaries.
    """
    merged = []
    for idx, source in enumerate(data_sources):
        for record in source:
            filtered_record = {k: v for k, v in record.items() if k in keys} if keys else record
            if len(merged) <= idx:
                merged.append({})
            merged[idx].update(filtered_record)
    return merged

if __name__ == "__main__":
    users = [
        {"id": 1, "name": "Alice", "email": "alice@example.com"},
        {"id": 2, "name": "Bob", "email": "bob@example.com"}
    ]
    orders = [
        {"id": 1, "product": "Laptop", "cost": 1200},
        {"id": 2, "product": "Keyboard", "cost": 75}
    ]
    result = merge_pipeline_data(users, orders, keys=("id", "name", "product", "cost"))
    print(json.dumps(result, indent=2))

Output

stdout
[
  {
    "id": 1,
    "name": "Alice",
    "product": "Laptop",
    "cost": 1200
  },
  {
    "id": 2,
    "name": "Bob",
    "product": "Keyboard",
    "cost": 75
  }
]

How it works

The merge_pipeline_data function iterates over each supplied data source and each record inside it. If a keys tuple is provided, only those keys are copied from the source dictionary; otherwise, all fields are kept. The positional index of the source determines the position of the merged dict in the final list, so sources are combined by row order. This pattern is common in data pipelines where related records from different tables must be joined by their index.

Common mistakes

  • Assuming sources are merged by a shared key like 'id' instead of by index
  • Not passing a keys tuple when you want only a subset of fields
  • Mixing sources of different lengths will drop the extra rows from the longer source

Variations

  1. Use pandas.merge to join by a key column when you need explicit join semantics
  2. Use dict comprehension with zip to merge two lists by index more concisely

Real-world use cases

  • Combining user and order CSV exports by row into a single analytics table
  • Blending output from multiple API endpoints into one denormalized report
  • Merging configuration dictionaries from separate environment files for a service

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Data pipelines & processing

Related tutorials and quizzes for this topic.