Attach Source File Metadata to Records in Python

Add a source filename field to each record in a list by merging a new key into every dictionary using a dict unpacking comprehension.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 15 views 0 copies

lineage metadata dict-unpacking pathlib etl comprehension

Python code

18 lines

Python 3.9+

from pathlib import Path
import json

def attach_source_metadata(records, source_file):
    """Attach source filename metadata to each record."""
    return [
        {**record, "source": Path(source_file).name}
        for record in records
    ]

if __name__ == "__main__":
    source = "/data/raw/customers.csv"
    sample_records = [
        {"id": 1, "name": "Alice"},
        {"id": 2, "name": "Bob"}
    ]
    enriched = attach_source_metadata(sample_records, source)
    print(json.dumps(enriched, indent=2))

Output

stdout

[
  {
    "id": 1,
    "name": "Alice",
    "source": "customers.csv"
  },
  {
    "id": 2,
    "name": "Bob",
    "source": "customers.csv"
  }
]

How it works

The {**record, "source": Path(source_file).name} expression creates a new dictionary with all existing fields plus a source key set to the basename of the file. Using Path(source_file).name extracts just the filename without the directory path, which keeps lineage metadata clean and portable. The list comprehension iterates over the original records without mutating them, returning a new enriched list — important when the source data should remain untouched for audit trails. This pattern is a common building block in ETL pipelines where each row needs a traceable origin.

Common mistakes

Mutating the original record with `record['source'] = ...` instead of returning a new dict
Using the full file path instead of `Path(source_file).name`, which breaks portability across environments
Forgetting that `**record` must come before the new key if you want the source field to override any pre-existing value

Variations

Use a `dataclass` with a `from_record` constructor for typed records
Add a timestamp alongside the source with `{**record, 'source': name, 'ingested_at': datetime.now().isoformat()}`
Attach file metadata lazily using a generator expression for streaming large datasets

Real-world use cases

Tagging records in a nightly data warehouse load with their raw source filename for lineage tracking and debugging.
Enriching rows pulled from multiple CSV exports with the originating file so downstream analytics can segment by source.
Adding provenance to log events or metrics batches so engineers can trace data quality issues back to the producing system.

Attach Source File Metadata to Records in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Keep learning

Tutorials

Quizzes