Compaction Small Files Mock in Python

Simulates a small-files compaction job by creating small mock files and merging them into a single output file using Python's standard library.

Easy Python 3.9+ Aug 9, 2026 Big data & Spark 16 views 0 copies

Python code

32 lines
Python 3.9+
from pathlib import Path
import tempfile
import os


def create_small_files(directory: Path, file_count: int = 5, lines_per_file: int = 3):
    """Create several small mock files with sample content."""
    directory.mkdir(exist_ok=True)
    for i in range(file_count):
        file_path = directory / f"part-{i:04d}.txt"
        content = "\n".join(f"row-{i}-{j}" for j in range(lines_per_file))
        file_path.write_text(content + "\n")


def compact_files(source_dir: Path, output_file: Path):
    """Merge all small files into a single compact file."""
    with output_file.open("w") as out_f:
        for file_path in sorted(source_dir.glob("*.txt")):
            out_f.write(file_path.read_text())


if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as tmp_dir:
        src_dir = Path(tmp_dir) / "source"
        create_small_files(src_dir, file_count=3, lines_per_file=2)

        output_path = Path(tmp_dir) / "compacted.txt"
        compact_files(src_dir, output_path)

        print(f"Small files: {len(list(src_dir.glob('*.txt')))}")
        print(f"Compacted content:")
        print(output_path.read_text())

Output

stdout
Small files: 3
Compacted content:
row-0-0
row-0-1
row-1-0
row-1-1
row-2-0
row-2-1

How it works

This script mimics the compaction of small files often needed in big data systems like Spark. The create_small_files function generates a set of small mock text files with deterministic content, simulating the output of a Spark job. compact_files reads each file in sorted order and writes their content into a single output file, handling the merge line by line. Using tempfile.TemporaryDirectory ensures the mock data is cleaned up automatically. Sorting the glob results is crucial to preserve a deterministic order in the compacted output.

Common mistakes

  • Forgetting to sort the file list from `glob`, leading to non-deterministic merge order.
  • Not ensuring the output file is closed properly; using a `with` block avoids resource leaks.
  • Assuming file content ends with a newline; the mock generation adds it explicitly.
  • Using `Path.read_text()` on huge files in production; here it's fine for small mocks.

Variations

  1. Use `os.scandir()` or `glob.glob()` with `key` for more control over file selection.
  2. If files are large, stream them line by line instead of `read_text()` to save memory.

Real-world use cases

  • Testing a compaction routine locally before applying it to actual Spark output partitions.
  • Simulating the merge of multiple small Parquet/CSV files into a single larger file for a data lake.
  • Validating that a compaction script preserves row order in a batch ETL pipeline.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Big data & Spark

Related tutorials and quizzes for this topic.