Test Python Automation Scripts

You've spent hours writing a Python automation script that provisions cloud resources, patches servers, or rotates credentials. You run it once, it works, and you deploy it to production. Then, two weeks later, it fails at 3 AM because a service returned an unexpected response or a config file changed format. The root cause? You never tested your automation script. In the DevOps world, automation is code, and code without tests is a liability. This lesson shows you how to test Python automation scripts so they behave predictably, fail loudly, and stay maintainable — even when the systems you automate are messy and dynamic.

The problem this lesson solves

Automation scripts in DevOps are not "throwaway" utilities. They orchestrate infrastructure, deploy applications, and handle sensitive data. When they fail, they can take down entire environments or leak credentials. Here's what goes wrong when you skip testing:

• Silent failures: A script that deploys a service might succeed locally but fail in CI because it assumes a specific Python version or environment variable.
• Breaking changes: The API of a cloud provider (like boto3) or a CLI tool (like kubectl) can change. Without tests, you won't notice until production breaks.
• Non-deterministic behavior: Scripts that rely on network calls, timeouts, or external state can behave differently each run. Testing forces you to make them deterministic.

The core problem is that automation scripts are often written in a REPL or written once and run forever, but they deserve the same rigor as application code. You need a way to validate logic, mock external dependencies, and simulate failure scenarios — without actually triggering disasters.

Pro tip: Testing isn't about proving your script works; it's about ensuring it fails safely when things go wrong.

Core concept / mental model

Think of a Python automation script as a function with side effects. The function gets inputs (config, environment variables, CLI args), does work (calls APIs, reads/writes files, runs commands), and produces outputs (return codes, log messages, or persistent changes). Testing means controlling the inputs, mocking the side-effect-heavy parts, and asserting on the outputs.

Here's the mental model:

• Pure logic (e.g., parsing config, calculating timeouts, building payloads) should be extracted into testable functions.
• External interactions (HTTP calls, subprocesses, file I/O) should be wrapped and mocked during tests.
• Entry points (CLI, scheduled job) should be thin and call the tested functions.

This separation is the key to test python automation scripts without needing live infrastructure.

How it works step by step

Let's build a testing strategy from scratch. Imagine you have a script that checks the health of a list of URLs and reports failures. Here's how you'd make it testable:

1. Separate logic from I/O: Create a function that decides if a response is healthy (status code, response time).
2. Use dependency injection: Pass a requests.Session or a generic "getter" function into your code instead of calling requests.get() directly.
3. Write tests with unittest or pytest: Use pytest for its simplicity and powerful fixtures.
4. Mock external calls: Use unittest.mock to simulate responses from the cloud API or subprocess output.
5. Test both success and failure paths: Simulate timeouts, bad status codes, and exceptions.
6. Make your script deterministic: For example, always use retries with exponential backoff, and make the retry count configurable (so tests can set it to 0).
7. Run tests automatically: Integrate them into CI (e.g., GitHub Actions, Jenkins) so every change is validated.

Each step builds on the last, and by the end you have a suite that catches regressions before they hit production.

Hands-on walkthrough

Let's write a small automation script and test it. We'll build a healthcheck.py that checks a list of services and logs any failures.

Step 1: Write the script with testability in mind

# healthcheck.py
import sys
import time
import requests

def is_healthy(response):
    """Return True if the response is healthy (2xx and fast enough)."""
    return response.status_code >= 200 and response.status_code < 300 and response.elapsed.total_seconds() < 2.0

def check_url(session, url):
    """Check a single URL, returning (url, ok, error)."""
    try:
        resp = session.get(url, timeout=5)
        ok = is_healthy(resp)
        return url, ok, None if ok else f"HTTP {resp.status_code}"
    except requests.RequestException as e:
        return url, False, str(e)

def main(urls, session=None):
    session = session or requests.Session()
    failures = []
    for url in urls:
        url, ok, err = check_url(session, url)
        if not ok:
            failures.append((url, err))
    if failures:
        print("FAILED services:", file=sys.stderr)
        for url, err in failures:
            print(f"  {url}: {err}", file=sys.stderr)
        return 1
    print("All services healthy")
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Step 2: Write tests using pytest and unittest.mock

# test_healthcheck.py
import pytest
from unittest.mock import Mock, patch
import requests
import healthcheck

def test_is_healthy_accepts_2xx():
    resp = Mock(status_code=200)
    resp.elapsed.total_seconds.return_value = 1.0
    assert healthcheck.is_healthy(resp) is True

def test_is_healthy_rejects_slow_response():
    resp = Mock(status_code=200)
    resp.elapsed.total_seconds.return_value = 2.5
    assert healthcheck.is_healthy(resp) is False

def test_check_url_returns_error_on_exception():
    session = Mock()
    session.get.side_effect = requests.ConnectionError("refused")
    url, ok, err = healthcheck.check_url(session, "http://example.com")
    assert ok is False
    assert "refused" in err

def test_main_returns_nonzero_on_failures(capsys):
    fake_session = Mock()
    fake_session.get.return_value = Mock(status_code=500, elapsed=Mock(total_seconds=0.1))
    result = healthcheck.main(["http://a", "http://b"], session=fake_session)
    assert result == 1
    captured = capsys.readouterr()
    assert "FAILED services" in captured.err

Run the tests with pytest -v. You'll see each test pass, verifying that your logic handles success, failures, and exceptions.

Step 3: Simulate real-world flakiness with retries

Let's add retry logic to make the script more resilient, and test that retries actually happen.

# healthcheck.py (additions)
def check_url_with_retries(session, url, retries=3, delay=1):
    for attempt in range(retries):
        url, ok, err = check_url(session, url)
        if ok:
            return url, True, None
        if attempt < retries - 1:
            time.sleep(delay)
    return url, False, err

def main(urls, session=None, retries=3):
    session = session or requests.Session()
    failures = []
    for url in urls:
        url, ok, err = check_url_with_retries(session, url, retries)
        if not ok:
            failures.append((url, err))
    # ... rest as before

Then test the retry behavior:

def test_retries_happen():
    session = Mock()
    session.get.return_value = Mock(status_code=500, elapsed=Mock(total_seconds=0.1))
    with patch("healthcheck.time.sleep") as mock_sleep:
        url, ok, err = healthcheck.check_url_with_retries(session, "http://a", retries=3, delay=1)
    assert ok is False
    assert session.get.call_count == 3
    mock_sleep.assert_called_with(1)

Now your test verifies that the script retries the right number of times and waits between attempts — without actually sleeping.

Compare options / when to choose what

When testing Python automation scripts, you have several tools and styles. Here's a quick comparison:

Choose pytest for almost everything because of its concise syntax and powerful fixtures. For AWS-heavy scripts, moto is a lifesaver. For scripts that shell out to kubectl or docker, mock subprocess.run and assert the command lines.

Pro tip: Always mock at the boundary where the external world meets your code — that keeps tests fast and deterministic.

Troubleshooting & edge cases

Even with tests, you'll hit issues. Here are common pitfalls and fixes:

• Test passes but script fails in production: Your mocks may be too perfect. Use realistic response objects (e.g., from responses) that include headers and text.
• Timing-dependent tests: Use freezegun to freeze time or mock time.sleep as we did above.
• Environment variables: Set them in fixtures using monkeypatch from pytest to avoid leaking state between tests.
• Network calls in setUp: You might accidentally hit the network in test setup. Always mock the session or use responses to intercept.
• Subprocess calls: If you script wraps kubectl, mock subprocess.run to return a CompletedProcess with stdout and returncode. Check that your script handles non-zero return codes.

Example of testing a subprocess-based script:

# deploy.py
import subprocess
def deploy(manifest):
    result = subprocess.run(["kubectl", "apply", "-f", manifest], capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError(result.stderr)
    return result.stdout

# test_deploy.py
def test_deploy_success(monkeypatch):
    def fake_run(*args, **kwargs):
        return subprocess.CompletedProcess(args[0], 0, stdout="deployment created", stderr="")
    monkeypatch.setattr(subprocess, "run", fake_run)
    assert deploy("pod.yaml") == "deployment created"

• When to skip tests: For one-off exploratory scripts, you might skip formal tests — but at least add assertions and run with python -m pytest to catch syntax errors.

What you learned & what's next

You've learned how to test Python automation scripts by structuring them as testable units, mocking external dependencies, and verifying success and failure paths. You now know how to:

• Explain why testing automation is non-negotiable in DevOps.
• Apply pytest and unittest.mock to your own scripts.
• Connect this skill to the broader track: the next lesson likely covers logging and monitoring for automation scripts, so your tests can verify that logs are emitted correctly. You're one step closer to building reliable, self-healing automation.

Final thought: A tested script is a script you can trust at 3 AM. Make testing a habit, and your automation will thank you.