How to Partition Output Files by Date Key in Python

Group output files into a dictionary partitioned by a YYYYMMDD date key extracted from the filename prefix.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 14 views 0 copies

file-partitioning date-key pathlib defaultdict file-organization

Python code

43 lines

Python 3.9+

from pathlib import Path
from collections import defaultdict

def partition_files_by_date(directory: str) -> dict:
    """Partition output files by date key extracted from filename (YYYYMMDD prefix)."""
    path = Path(directory)
    partitions = defaultdict(list)
    
    for file in path.iterdir():
        if file.is_file():
            name = file.name
            # Extract date key from filename: YYYYMMDD_rest.txt
            if len(name) >= 8 and name[:8].isdigit():
                date_key = name[:8]
                partitions[date_key].append(name)
    
    return dict(partitions)

# Example usage
if __name__ == "__main__":
    # Create sample files for demonstration
    import tempfile
    import os
    
    with tempfile.TemporaryDirectory() as tmpdir:
        sample_files = [
            "20240101_report.txt",
            "20240101_data.csv",
            "20240102_summary.txt",
            "20240103_notes.txt",
            "20240103_log.txt",
            "README.md"  # This one won't be included
        ]
        
        for fname in sample_files:
            filepath = os.path.join(tmpdir, fname)
            with open(filepath, "w") as f:
                f.write("sample content")
        
        result = partition_files_by_date(tmpdir)
        
        for date, files in sorted(result.items()):
            print(f"{date}: {files}")

Output

stdout

20240101: ['20240101_report.txt', '20240101_data.csv']
20240102: ['20240102_summary.txt']
20240103: ['20240103_notes.txt', '20240103_log.txt']

How it works

The function uses Path.iterdir() to list all entries in the directory, filtering out non-files with is_file(). For each filename, it checks if the first 8 characters are digits, which indicates a valid date key, and then appends the filename to the corresponding list in a defaultdict. Using defaultdict(list) avoids manual key checking and makes the code concise. Finally, converting to a regular dict yields a clean structure for downstream processing.

Common mistakes

Not checking that exactly 8 leading characters are digits; files like 'report20240101.txt' will be skipped.
Forgetting `file.is_file()` and including subdirectories in the partition.
Assuming the date key is always the prefix without validating with `.isdigit()`.
Using `os.listdir` and manually joining paths when `Path.iterdir` is cleaner.

Variations

Use `collections.defaultdict(set)` if you need unique filenames or want to deduplicate.
Extract the date using regex `re.match(r'(\d{8})_', name)` for stricter pattern matching.

Real-world use cases

Organizing daily log files generated by a service into folders per date for archival.
Partitioning exported data files by ingestion date before loading into a data warehouse.
Grouping media uploads by the day they were captured for batch processing pipelines.

How to Partition Output Files by Date Key in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Keep learning

Tutorials

Quizzes