How to Split a Large File into Fixed-Size Parts in Python
Splits any binary or text file into multiple part files of a fixed byte size using Python's standard library.
Python code
31 linesimport os
import math
def split_file(filepath, chunk_size_bytes):
"""Split a file into parts of fixed size (bytes). Creates part files in same directory."""
filepath = os.path.abspath(filepath)
basename = os.path.basename(filepath)
file_size = os.path.getsize(filepath)
num_parts = math.ceil(file_size / chunk_size_bytes)
with open(filepath, 'rb') as f:
for part_num in range(num_parts):
chunk = f.read(chunk_size_bytes)
part_filename = f"{basename}.part{part_num + 1:03d}"
with open(part_filename, 'wb') as part_file:
part_file.write(chunk)
print(f"Created {part_filename} ({len(chunk)} bytes)")
if __name__ == "__main__":
# Example: split a sample file into 16-byte chunks
sample = "example_data.txt"
with open(sample, 'w') as f:
f.write("Hello world, this is a test file for splitting.")
split_file(sample, 16)
# Cleanup after demonstration
for fname in os.listdir('.'):
if fname.startswith(sample) and fname != sample:
os.remove(fname)
os.remove(sample)
Output
Created example_data.txt.part001 (16 bytes)
Created example_data.txt.part002 (16 bytes)
Created example_data.txt.part003 (11 bytes)
How it works
The function uses os.path.getsize to determine the file size and math.ceil to calculate the number of parts. Reading in binary mode with 'rb' ensures that the splitting works for both text and binary files without any encoding issues. Each chunk is read with f.read(chunk_size_bytes), which returns exactly that many bytes except for the last part, which contains the remaining data. Part files are named with a zero-padded index to maintain order, and the script cleans up after the demo by removing the created parts and the original sample file.
Common mistakes
- Forgetting to use binary mode ('rb'/'wb') when splitting text files, leading to encoding errors
- Not padding part numbers (e.g., part1, part2) which breaks alphabetical ordering for more than 9 parts
- Assuming the last chunk is the same size; it is usually smaller
- Overwriting existing part files without checking, which can lose data
Variations
- Use `shutil.copyfileobj` with a loop for memory-efficient large-file splitting
- Store parts in a subdirectory to keep the original directory clean
Real-world use cases
- Splitting a large database dump into chunks small enough to upload to an object store with per-file size limits.
- Breaking a log file into daily parts for easier archival and transfer between systems.
- Segmenting a video or archive into pieces for distribution across multiple HTTP requests (e.g., resumable downloads).
Sponsored
More from Files & data
- Append a Line to a Log File in Python easy
- Audit File Permissions Across a Project in Python easy
- Automatically Detect Corrupted Files Using SHA-256 Checksums in Python easy
- Automatically Highlight Data Validation Errors Inside Excel Files in Python easy
- Build a Command-Line To-Do List Application with Data Persistence in Python easy
- Build a File Index by Relative Path Hash Map in Python easy
Keep learning
Related tutorials and quizzes for this topic.