Build a Health Check Tool

Your monitoring dashboard is only as good as the data it gets — and that data comes from health checks. A single down service can cascade into a full-blown incident, yet manually curling endpoints is slow, error-prone, and doesn't scale. In this lesson, you're going to create a Python-based health check tool that pings HTTP endpoints, measures response times, and reports status — the exact kind of script every DevOps engineer needs in their toolkit.

The problem this lesson solves

When you run a fleet of services — APIs, web apps, databases behind a load balancer — you need to know the moment one goes down. Manually checking each endpoint with curl or a browser is fine for a one-off, but it's impossible to do continuously across dozens of services. Worse, intermittent failures (a 500 error that happens once every hour) are easy to miss.

A health check tool automates this: it periodically sends requests to your services, checks the status code, measures latency, and logs the result. It's the foundation of any uptime monitoring system, and it's often the first script a DevOps engineer writes. Without it, you're flying blind — and in production, that's dangerous.

Real-world impact: A health check that runs every 30 seconds can catch a downed service within a minute, while manual checking might take 10+ minutes — that's a 10x reduction in mean time to detection.

Core concept / mental model

Think of a health check tool as a digital stethoscope for your services. It listens to each endpoint, interprets the response (a heartbeat), and tells you if the patient is alive. But instead of a single check, you're monitoring many patients at once.

The core loop is simple:

1. Define a list of endpoints to check (URL, expected status code, timeout).
2. For each endpoint, send a request (GET, HEAD, or POST).
3. Evaluate the response: is the status code as expected? Did it come back within the timeout?
4. Record status: up, down, or degraded (e.g., slow response).
5. Output the results in a structured format (JSON, CSV, plain text) for logging or alerting.

A common analogy: a smoke detector. It constantly watches for smoke (failure) and only alerts when something's wrong. Your health check tool does the same — but with endpoints instead of smoke, and with configurable thresholds.

Key terms

• Endpoint: A URL that returns a health status (e.g., /health, /ready, /live).
• Status code: HTTP response code (200 OK, 404 Not Found, 500 Server Error).
• Timeout: Maximum time to wait for a response before considering the service down.
• Latency: Time it takes for the request to complete — high latency can indicate degradation.

How it works step by step

Let's break down the process of building the tool. Each step builds on the last, so follow the sequence carefully.

Step 1: Structure the script

Start with a list of health check targets. You can hardcode them as dictionaries or load from a config file (e.g., JSON). Each entry should include:

• name: human-readable identifier
• url: the endpoint to check
• expected_status: what status code indicates healthy (usually 200)
• timeout: seconds to wait before failing
• max_latency: optional threshold for degraded performance

Step 2: Perform the check

Use requests.get() (or HEAD for lighter checks) to fetch the URL. Catch exceptions like ConnectionError and Timeout. Record the elapsed time.

Step 3: Evaluate health

Compare the actual status code to expected. If they differ, it's down. If response time exceeds max_latency, mark as degraded.

Step 4: Record and output

Collect results in a list of dictionaries, then write to stdout (human-readable) or JSON (machine-readable). For automation, JSON is ideal because other tools (like Dashboards) can consume it.

Step 5: Make it reusable

Wrap the logic in functions so you can easily extend it to different endpoints or add alerting. Use if __name__ == '__main__' to keep it runnable as a script.

Hands-on walkthrough

Let's implement a fully functional version. We'll use requests and concurrent.futures to check multiple endpoints in parallel.

Example 1: Basic single-check function

import requests

def check_health(url: str, expected_status: int = 200, timeout: int = 5):
    try:
        response = requests.get(url, timeout=timeout)
        return {
            'url': url,
            'status_code': response.status_code,
            'healthy': response.status_code == expected_status,
            'latency_ms': round(response.elapsed.total_seconds() * 1000, 2)
        }
    except requests.exceptions.RequestException as e:
        return {
            'url': url,
            'status_code': None,
            'healthy': False,
            'error': str(e)
        }

result = check_health('https://jsonplaceholder.typicode.com/posts/1')
print(result)

Expected output (example):

{'url': 'https://jsonplaceholder.typicode.com/posts/1', 'status_code': 200, 'healthy': True, 'latency_ms': 180.23}

Example 2: Checking multiple endpoints concurrently

import requests
import concurrent.futures

endpoints = [
    {'name': 'api', 'url': 'https://jsonplaceholder.typicode.com/posts', 'expected': 200, 'timeout': 5},
    {'name': 'images', 'url': 'https://httpstat.us/404', 'expected': 404, 'timeout': 5},
    {'name': 'slow', 'url': 'https://httpstat.us/200?sleep=3000', 'expected': 200, 'timeout': 5},
]

def check_one(ep: dict):
    try:
        r = requests.get(ep['url'], timeout=ep['timeout'])
        return {
            'name': ep['name'],
            'url': ep['url'],
            'status_code': r.status_code,
            'healthy': r.status_code == ep['expected'],
            'latency_ms': round(r.elapsed.total_seconds() * 1000, 2)
        }
    except Exception as e:
        return {'name': ep['name'], 'healthy': False, 'error': str(e)}

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    results = list(executor.map(check_one, endpoints))

for res in results:
    print(res)

Expected output (truncated):

{'name': 'api', 'healthy': True, 'latency_ms': 85.0}
{'name': 'images', 'healthy': True, 'status_code': 404, 'latency_ms': 42.0}
{'name': 'slow', 'healthy': True, 'latency_ms': 3010.0}

Example 3: Retry logic and degraded status

import time

def check_with_retry(url: str, expected: int = 200, timeout: int = 5, retries: int = 2):
    for attempt in range(retries + 1):
        try:
            r = requests.get(url, timeout=timeout)
            return {
                'url': url,
                'status_code': r.status_code,
                'healthy': r.status_code == expected,
                'attempt': attempt + 1
            }
        except requests.exceptions.Timeout:
            if attempt == retries:
                return {'url': url, 'healthy': False, 'error': 'timeout after retries'}
            time.sleep(1)
        except requests.exceptions.ConnectionError as e:
            if attempt == retries:
                return {'url': url, 'healthy': False, 'error': str(e)}
            time.sleep(1)

check = check_with_retry('https://httpstat.us/503')
print(check)

Expected output: Depending on the service, it might return healthy=False if the endpoint returns 503 or raises an error.

Example 4: Writing results to a file

import json

results = [
    {'name': 'api', 'healthy': True, 'latency_ms': 85.0},
    {'name': 'db', 'healthy': False, 'error': 'connection refused'}
]

with open('health_results.json', 'w') as f:
    json.dump(results, f, indent=2)

print("Results written to health_results.json")

Compare options / when to choose what

Pro tip: For 90% of cases, requests + ThreadPoolExecutor is the sweet spot. It's fast, readable, and easy to extend. If you need zero dependencies, fall back to urllib or http.client.

Troubleshooting & edge cases

• requests.exceptions.SSLError — If your service uses self-signed certs, add verify=False (with caution) or verify='/path/to/cert.pem'.
• Timeout not honored — If you're using json payloads, the connection timeout is separate from the read timeout. Increase timeout=(connect, read).
• False positives from redirects — Some endpoints redirect to login pages (302). Set allow_redirects=False to prevent following redirects if the expected status is different.
• Zero latency reported — If you measure elapsed before fully reading the response body, you may miss time spent downloading. Ensure you read response.content before measuring.
• Thread safety in logs — When writing to a file from multiple threads, use a lock to avoid interleaved writes.
• Non-200 expected statuses — Some endpoints return 404 or 500 for health checks (e.g., /health returning 200 only when ready). Always set expected accordingly.
• Geographical latency — If your check runs from a remote region, latency might be high legitimately. Set max_latency with context, not globally.

What you learned & what's next

You've now built a Python-based health check tool that checks multiple endpoints concurrently, handles retries, and outputs structured results. You understand the mental model of health checks as digital stethoscopes, and you can compare different implementation approaches. You've also mastered the key edge cases: SSL, timeouts, redirects, and concurrency.

You are ready to handle real-world scenarios like monitoring microservices, validating deployments, and feeding data into dashboards.

Next lesson: In the next step of your Python for DevOps automation learning path, we'll explore integrating health checks with alerting — how to send notifications (Slack, email) when a service goes down, and how to parse the JSON output into scheduling tools like Cron. This turns your simple checker into a proactive monitoring system.

Now go ahead and run the examples — then try extending the tool with your own endpoints and thresholds. Happy monitoring!