Compaction Small Files Mock in Python
Simulates a small-files compaction job by creating small mock files and merging them into a single output file using Python's standard library.
Python code
32 linesfrom pathlib import Path
import tempfile
import os
def create_small_files(directory: Path, file_count: int = 5, lines_per_file: int = 3):
"""Create several small mock files with sample content."""
directory.mkdir(exist_ok=True)
for i in range(file_count):
file_path = directory / f"part-{i:04d}.txt"
content = "\n".join(f"row-{i}-{j}" for j in range(lines_per_file))
file_path.write_text(content + "\n")
def compact_files(source_dir: Path, output_file: Path):
"""Merge all small files into a single compact file."""
with output_file.open("w") as out_f:
for file_path in sorted(source_dir.glob("*.txt")):
out_f.write(file_path.read_text())
if __name__ == "__main__":
with tempfile.TemporaryDirectory() as tmp_dir:
src_dir = Path(tmp_dir) / "source"
create_small_files(src_dir, file_count=3, lines_per_file=2)
output_path = Path(tmp_dir) / "compacted.txt"
compact_files(src_dir, output_path)
print(f"Small files: {len(list(src_dir.glob('*.txt')))}")
print(f"Compacted content:")
print(output_path.read_text())
Output
Small files: 3
Compacted content:
row-0-0
row-0-1
row-1-0
row-1-1
row-2-0
row-2-1
How it works
This script mimics the compaction of small files often needed in big data systems like Spark. The create_small_files function generates a set of small mock text files with deterministic content, simulating the output of a Spark job. compact_files reads each file in sorted order and writes their content into a single output file, handling the merge line by line. Using tempfile.TemporaryDirectory ensures the mock data is cleaned up automatically. Sorting the glob results is crucial to preserve a deterministic order in the compacted output.
Common mistakes
- Forgetting to sort the file list from `glob`, leading to non-deterministic merge order.
- Not ensuring the output file is closed properly; using a `with` block avoids resource leaks.
- Assuming file content ends with a newline; the mock generation adds it explicitly.
- Using `Path.read_text()` on huge files in production; here it's fine for small mocks.
Variations
- Use `os.scandir()` or `glob.glob()` with `key` for more control over file selection.
- If files are large, stream them line by line instead of `read_text()` to save memory.
Real-world use cases
- Testing a compaction routine locally before applying it to actual Spark output partitions.
- Simulating the merge of multiple small Parquet/CSV files into a single larger file for a data lake.
- Validating that a compaction script preserves row order in a batch ETL pipeline.
Sponsored
More from Big data & Spark
- Accumulators Global Counter Mock in Python medium
- Approximate Distinct Count in Python with HyperLogLog medium
- Bloom Filter Join Mock in Python medium
- Cache persist MEMORY_ONLY mock in Python easy
- Delta Lake ACID Transaction Log Mock in Python medium
- How to Broadcast a Small Lookup Table in Python easy
Keep learning
Related tutorials and quizzes for this topic.