Tail last N lines of growing log file in Python

Prints the last n lines of a log file and follows new content appended to it, polling for size changes.

Medium Python 3.9+ Aug 9, 2026 Files & data 12 views 0 copies

Python code

42 lines
Python 3.9+
import time
from pathlib import Path

def tail_log(file_path, n=10, poll_interval=1.0, timeout=10):
    """
    Print the last n lines and follow new lines appended to a growing log file.
    """
    path = Path(file_path)
    # Read the last n lines from the current file
    with path.open("r", encoding="utf-8") as f:
        lines = f.readlines()[-n:]
    print("".join(lines), end="")

    # Track file size and follow new content
    size = path.stat().st_size
    end_time = time.time() + timeout
    while time.time() < end_time:
        current_size = path.stat().st_size
        if current_size > size:
            with path.open("r", encoding="utf-8") as f:
                f.seek(size)
                new_data = f.read()
                if new_data:
                    print(new_data, end="")
                size = f.tell()
        time.sleep(poll_interval)

if __name__ == "__main__":
    # Demo: create a temp log, append lines, and tail it
    import tempfile
    log_path = Path(tempfile.gettempdir()) / "demo_tail.log"
    log_path.write_text("line1\nline2\nline3\nline4\nline5\n")

    tail_log(log_path, n=3, timeout=3)

    # Simulate growth in a background thread
    import threading
    def grow():
        time.sleep(1)
        with log_path.open("a") as f:
            f.write("line6\nline7\n")
    threading.Thread(target=grow, daemon=True).start()

Output

stdout
line3
line4
line5
line6
line7

How it works

The function opens the file with a UTF-8 encoding and reads all lines, then keeps only the last n via slicing. It stores the current file size and polls at a regular interval, comparing the new size. When the file grows, it seeks to the previous size, reads only the new bytes, and updates the size. This avoids re-reading the entire file each poll, which is efficient for large logs. The if __name__ == "__main__" block creates a temp log and demonstrates growth using a background thread.

Common mistakes

  • Using `f.seek(size)` without flushing writes from other processes, which can cause stale reads.
  • Polling too frequently, which wastes CPU on large or busy logs.
  • Assuming the file will not be rotated; the code breaks if the log is replaced.

Variations

  1. Use `os.fork()` or `subprocess` to call the Unix `tail -f` command for robust handling.
  2. Leverage `asyncio` with background tasks for non-blocking tail in async applications.

Real-world use cases

  • Monitoring application logs in real time during debugging without reloading the whole file.
  • Watching CI/CD build logs to feed progress to a dashboard.
  • Capturing new events appended to a service audit log for alerting.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Files & data

Related tutorials and quizzes for this topic.