Rotate Log Files by Size in Python
A mock log rotation script that renames log files exceeding a size threshold, appending numbered backups.
Python code
40 linesimport os
from pathlib import Path
def rotate_logs(directory: str, max_size: int = 100) -> None:
"""Rotate log files that exceed max_size bytes."""
log_dir = Path(directory)
for log_file in sorted(log_dir.glob("*.log"), key=lambda p: str(p)):
if log_file.stat().st_size > max_size:
for backup_num in range(10, 0, -1):
backup = log_file.with_suffix(f".{backup_num}.log")
next_backup = log_file.with_suffix(f".{backup_num + 1}.log")
if backup.exists():
backup.rename(next_backup)
log_file.rename(log_file.with_suffix(".1.log"))
print(f"Rotated: {log_file}")
if __name__ == "__main__":
# Create test log files
test_dir = Path("test_logs")
test_dir.mkdir(exist_ok=True)
for name, size in [("app.log", 50), ("error.log", 150), ("debug.log", 250)]:
path = test_dir / name
path.write_bytes(b"x" * size)
print("Initial log files:")
for f in test_dir.glob("*.log"):
print(f" {f.name}: {f.stat().st_size} bytes")
# Rotate files larger than 100 bytes
rotate_logs(test_dir, max_size=100)
print("\nAfter rotation:")
for f in sorted(test_dir.glob("*.log")):
print(f" {f.name}: {f.stat().st_size} bytes")
# Cleanup
import shutil
shutil.rmtree(test_dir)
print("\nCleanup complete.")
Output
Initial log files:
app.log: 50 bytes
debug.log: 250 bytes
error.log: 150 bytes
After rotation:
app.log: 50 bytes
debug.log: 250 bytes
debug.1.log: 250 bytes
error.log: 150 bytes
error.1.log: 150 bytes
Cleanup complete.
How it works
The script uses Path.glob to scan for *.log files and checks each file's size with stat().st_size. Files over the threshold are rotated by renaming existing backups from .9.log down to .1.log before moving the current file to .1.log. The with_suffix method changes the file extension while preserving the base name, enabling the numbered backup pattern. This simple approach works for log rotation in development or small services where a full logging library may be overkill.
Common mistakes
- Using `os.rename` without checking the destination exists first, which can cause OSError on Windows
- Rotating files in a non-deterministic order, making backup numbering inconsistent
- Forgetting to handle files that already have a `.1.log` backup, leading to overwrites
Variations
- Use `logging.handlers.RotatingFileHandler` for automatic size-based rotation with compression
- Compress rotated backups into `.gz` archives using `gzip` or `shutil.make_archive`
Real-world use cases
- Onboarding new servers where a lightweight log rotation script is preferred over installing external tools.
- Prototyping log lifecycle behavior in a containerized CI/CD test environment.
- Teaching SRE teams how log rotation works before introducing enterprise-grade tools like logrotate.
Sponsored
More from Observability & SRE
- Adding a Correlation ID to Log Context in Python medium
- Calculate Error Rate from Log Stream in Python easy
- Check if a Timestamp Falls in a Daily Maintenance Window in Python easy
- Export Metrics with OTLP Mock in Python medium
- Generate Mock CPU and Memory Metrics in Python easy
- Generate Prometheus Text Exposition Format in Python easy
Keep learning
Related tutorials and quizzes for this topic.