Tail last N lines of growing log file in Python
Prints the last n lines of a log file and follows new content appended to it, polling for size changes.
Python code
42 linesimport time
from pathlib import Path
def tail_log(file_path, n=10, poll_interval=1.0, timeout=10):
"""
Print the last n lines and follow new lines appended to a growing log file.
"""
path = Path(file_path)
# Read the last n lines from the current file
with path.open("r", encoding="utf-8") as f:
lines = f.readlines()[-n:]
print("".join(lines), end="")
# Track file size and follow new content
size = path.stat().st_size
end_time = time.time() + timeout
while time.time() < end_time:
current_size = path.stat().st_size
if current_size > size:
with path.open("r", encoding="utf-8") as f:
f.seek(size)
new_data = f.read()
if new_data:
print(new_data, end="")
size = f.tell()
time.sleep(poll_interval)
if __name__ == "__main__":
# Demo: create a temp log, append lines, and tail it
import tempfile
log_path = Path(tempfile.gettempdir()) / "demo_tail.log"
log_path.write_text("line1\nline2\nline3\nline4\nline5\n")
tail_log(log_path, n=3, timeout=3)
# Simulate growth in a background thread
import threading
def grow():
time.sleep(1)
with log_path.open("a") as f:
f.write("line6\nline7\n")
threading.Thread(target=grow, daemon=True).start()
Output
line3
line4
line5
line6
line7
How it works
The function opens the file with a UTF-8 encoding and reads all lines, then keeps only the last n via slicing. It stores the current file size and polls at a regular interval, comparing the new size. When the file grows, it seeks to the previous size, reads only the new bytes, and updates the size. This avoids re-reading the entire file each poll, which is efficient for large logs. The if __name__ == "__main__" block creates a temp log and demonstrates growth using a background thread.
Common mistakes
- Using `f.seek(size)` without flushing writes from other processes, which can cause stale reads.
- Polling too frequently, which wastes CPU on large or busy logs.
- Assuming the file will not be rotated; the code breaks if the log is replaced.
Variations
- Use `os.fork()` or `subprocess` to call the Unix `tail -f` command for robust handling.
- Leverage `asyncio` with background tasks for non-blocking tail in async applications.
Real-world use cases
- Monitoring application logs in real time during debugging without reloading the whole file.
- Watching CI/CD build logs to feed progress to a dashboard.
- Capturing new events appended to a service audit log for alerting.
Sponsored
More from Files & data
- Append a Line to a Log File in Python easy
- Audit File Permissions Across a Project in Python easy
- Automatically Detect Corrupted Files Using SHA-256 Checksums in Python easy
- Automatically Highlight Data Validation Errors Inside Excel Files in Python easy
- Build a Command-Line To-Do List Application with Data Persistence in Python easy
- Build a File Index by Relative Path Hash Map in Python easy
Keep learning
Related tutorials and quizzes for this topic.