Automate File System Tasks

Manually managing files across dozens of servers, logs, and backups is a recipe for burnout — and a prime source of human error in any DevOps pipeline. Every deployment, every cron job, every data backup ultimately boils down to file system operations: creating, moving, copying, cleaning up. When you automate file system tasks with Python, you turn repetitive, error-prone shell commands into deterministic, testable, and repeatable code — the core of every reliable infrastructure workflow.

The Problem This Lesson Solves

You've probably scripted file operations with a patchwork of cp, mv, rm, and find commands glued together in Bash. That approach works — until it doesn't. A single typo in a recursive rm -rf can wipe out an entire environment. Shell scripts are brittle: no native error handling, no type safety, and they behave differently across Linux, macOS, and Windows. As your infrastructure grows, you need something more robust.

Consider the typical DevOps pain points:

• Deployment artifacts piling up in unexpected locations
• Log rotation that never runs at the right time
• Backup verification that silently fails
• Cross-platform consistency that breaks when you move from Linux to Windows

Python's standard library gives you battle-tested modules like os, shutil, and pathlib to automate file system tasks safely and portably. You get structured error handling, path abstraction, and the ability to compose complex workflows that are readable and maintainable.

Core Concept / Mental Model

Think of the file system as a tree — directories are branches, files are leaves. Automating file system tasks means writing code that navigates and manipulates that tree programmatically. Instead of manually typing commands, you give Python a set of instructions: walk the branches, find matching leaves, apply an operation, and log what you did.

Three core modules form the foundation:

• pathlib (modern, object-oriented paths)
• os (low-level file operations)
• shutil (high-level file operations like copying and moving)

The pathlib Advantage

Introduced in Python 3.4, pathlib treats paths as first-class objects. You can chain methods, combine paths with the / operator, and avoid countless string concatenation bugs.

Pro tip: Always use pathlib.Path over raw strings for file paths. It's cleaner, more readable, and handles platform differences automatically.

How It Works Step by Step

Automating file system tasks follows a predictable pattern:

1. Define the source and destination paths — use pathlib.Path for portability.
2. Check existence and permissions — guard your operations.
3. Perform the core operation — create, copy, move, delete, or traverse.
4. Handle errors gracefully — catch exceptions like FileNotFoundError and PermissionError.
5. Log and report — track what happened for auditing and debugging.

Let's break each step with concrete examples.

Creating Directories

from pathlib import Path

base_dir = Path("/tmp/my_project")
logs_dir = base_dir / "logs"
backup_dir = base_dir / "backups/2025-01-15"

# Create directories recursively (ignore if they already exist)
backup_dir.mkdir(parents=True, exist_ok=True)
logs_dir.mkdir(parents=True, exist_ok=True)

print(f"Backup directory: {backup_dir}")
print(f"Logs directory: {logs_dir}")

Expected output:

Backup directory: /tmp/my_project/backups/2025-01-15
Logs directory: /tmp/my_project/logs

Copying and Moving Files

shutil provides high-level functions that handle copy operations reliably. The key difference between copy2 and move is that copy2 preserves metadata while move can move across filesystems.

import shutil
from pathlib import Path

src = Path("report.csv")
dst = Path("/backups/report.csv")

# Copy (preserves metadata)
shutil.copy2(src, dst)
print(f"Copied {src} -> {dst}")

# Move (can cross filesystem boundaries)
shutil.move(src, dst)
print(f"Moved {src} -> {dst}")

Expected output:

Copied report.csv -> /backups/report.csv
Moved report.csv -> /backups/report.csv

Walking a Directory Tree

Often you need to process every file in a directory tree — for example, to archive logs or clean up old temp files. pathlib.Path.rglob() is a generator that lazily yields matching paths.

from pathlib import Path

log_dir = Path("./logs")

# Find all .log files recursively
for log_file in log_dir.rglob("*.log"):
    print(f"Found: {log_file}")
    # You could archive, compress, or delete it

Expected output (assuming you have logs):

Found: logs/app.log
Found: logs/nginx/access.log

Deleting Files and Directories

Be extremely careful here — deletion is irreversible. Always use safety checks and consider a 'dry run' mode.

import shutil
from pathlib import Path

target_dir = Path("/tmp/old_builds")

if not target_dir.exists():
    print("Directory not found, nothing to delete.")
elif "danger" in str(target_dir):
    print("Skipping deletion due to safety flag.")
else:
    shutil.rmtree(target_dir)
    print(f"Deleted {target_dir}")

Hands-On Walkthrough

Let's build a real-world script that automates a daily backup task — a common DevOps chore. The script will:

• Copy all .csv files from a source directory to a timestamped backup folder
• Compress backups older than 7 days into a .tar.gz archive
• Delete archives older than 30 days

Step 1: Folder Structure

Create these directories to test:

mkdir -p /tmp/devops_demo/source
mkdir -p /tmp/devops_demo/backups

Place a few .csv files in source.

Step 2: The Backup Script

import shutil
import tarfile
from pathlib import Path
from datetime import datetime, timedelta

SOURCE_DIR = Path("/tmp/devops_demo/source")
BACKUP_ROOT = Path("/tmp/devops_demo/backups")
RETENTION_DAYS = 30
COMPRESS_AFTER_DAYS = 7

# Create backup root if missing
BACKUP_ROOT.mkdir(exist_ok=True)

# 1. Create a timestamped backup folder for today
today_str = datetime.now().strftime("%Y-%m-%d")
today_backup = BACKUP_ROOT / today_str
today_backup.mkdir(exist_ok=False)  # Fail if exists to avoid overwrites

# 2. Copy all .csv files
csv_count = 0
for csv_file in SOURCE_DIR.glob("*.csv"):
    shutil.copy2(csv_file, today_backup / csv_file.name)
    csv_count += 1
print(f"Copied {csv_count} CSV files to {today_backup}")

# 3. Compress backups older than 7 days
cutoff = datetime.now() - timedelta(days=COMPRESS_AFTER_DAYS)
for folder in BACKUP_ROOT.iterdir():
    if not folder.is_dir():
        continue
    try:
        folder_date = datetime.strptime(folder.name, "%Y-%m-%d")
    except ValueError:
        print(f"Skipping non-date folder: {folder.name}")
        continue
    if folder_date < cutoff and not (folder.with_suffix(".tar.gz")).exists():
        archive_name = folder.name + ".tar.gz"
        with tarfile.open(BACKUP_ROOT / archive_name, "w:gz") as tar:
            tar.add(folder, arcname=folder.name)
        shutil.rmtree(folder)
        print(f"Compressed {folder.name} into {archive_name}")

# 4. Delete archives older than 30 days
cutoff_delete = datetime.now() - timedelta(days=RETENTION_DAYS)
for archive in BACKUP_ROOT.glob("*.tar.gz"):
    try:
        archive_date = datetime.strptime(archive.stem, "%Y-%m-%d")
    except ValueError:
        continue
    if archive_date < cutoff_delete:
        archive.unlink()
        print(f"Deleted old archive: {archive.name}")

Expected Output (first run)

Copied 3 CSV files to /tmp/devops_demo/backups/2025-01-15

On subsequent runs (if you wait 7 days), you'd see compression and deletion messages. This script is deterministic but requires manual cron scheduling — you'll automate that in a later lesson.

Step 3: Extend to Archive Old Logs

You can easily extend the same pattern to clean up old logs. For example, move files with .log extension older than N days to an archive folder. The key is reusing the same pathlib and shutil building blocks.

Compare Options / When to Choose What

When automating file system tasks, you'll often choose between os vs pathlib, and copy vs move. Here's a quick comparison:

When to use what:

• For path manipulation and existence checks, prefer pathlib.Path.
• For copying files, shutil.copy2 is your go-to.
• For moving files, shutil.move is safest.
• For bulk operations with filters, combine pathlib.rglob() with shutil.

Pro tip: Avoid os.path unless you're stuck with legacy code. pathlib is the future-proof choice.

Troubleshooting & Edge Cases

Even with best practices, things go wrong. Here are the most common issues and how to fix them.

FileNotFoundError

Symptom: FileNotFoundError or OSError. Fix: Always check existence before operating.

from pathlib import Path

path = Path("/tmp/missing/file.txt")
if path.exists():
    path.unlink()
else:
    print("File does not exist, skipping.")

PermissionError

Symptom: PermissionError: [Errno 13] Permission denied. Fix: Verify permissions, run with appropriate privileges, or handle gracefully.

import os
from pathlib import Path

path = Path("/protected/file.txt")
try:
    path.unlink()
except PermissionError:
    print(f"No permission to delete {path}. Check user or sudo.")

Directory Not Empty

When using os.rmdir(), the directory must be empty. Use shutil.rmtree() for non-empty dirs, but be aware it deletes recursively.

Windows Path Differences

On Windows, paths use backslashes and drive letters. pathlib handles this automatically, so avoid hardcoding / or \.

Accidental Deletion

Add safety checks before destructive operations:

import shutil
from pathlib import Path

# Use a dummy flag for dry-run
DRY_RUN = True

if DRY_RUN:
    print(f"Would delete {target_dir}")
else:
    shutil.rmtree(target_dir)

Symlink Loops

When walking directories, symlinks can cause infinite loops. Use Path.is_file() (which follows symlinks) carefully, or use os.walk(followlinks=False).

What You Learned & What's Next

You've now mastered the core of automating file system tasks in Python:

• The mental model: file systems as trees, with Python modules to navigate and manipulate them.
• The essential modules: pathlib, os, shutil.
• Practical patterns: create, copy, move, delete, walk, and compress files.
• Troubleshooting: handling missing files, permissions, and cross-platform quirks.
• Safety: using dry-runs and checks to prevent data loss.

You can now apply these skills to real-world scenarios like log rotation, backup management, and artifact cleanup.

In the next lesson, you'll learn how to automate deployments by combining file system operations with remote execution tools — taking your automation to the next level.

Keep practicing: try modifying the backup script to handle .json files or to send a summary email when backups complete. The more you automate, the more time you free up for higher-value work.