Container health checks

Ever deployed a container that seemed healthy but silently stopped serving traffic? That's the pain this lesson solves. Without a health check, Docker only knows if the container's main process is running, not if your app is actually responding correctly. You'll learn to manage container health checks so your orchestrator (and you) can detect and restart truly broken containers automatically.

The problem this lesson solves

A running container doesn't mean a working application. A web server might be alive but hung on a deadlock, a database process might be running but refusing connections, or an API might be crash-looping faster than Docker's default restart policy can catch. The core problem: Docker's default view is process-centric (PID 1 alive? Container is up). Production systems need application-centric awareness — "is my service healthy enough to handle traffic?"

Without manage container health checks, you'll: - Restart containers that are fine, or fail to restart containers that are silently broken. - Waste time debugging "running" containers that don't serve requests. - Leave orchestration tools (Docker Swarm, Compose, Kubernetes) blind to real failures.

Core concept / mental model

Think of a health check as a contract signed by your container. Every few seconds, Docker runs a command inside the container. If that command succeeds (exit code 0), the container is marked healthy. If it fails repeatedly, Docker marks it unhealthy and can auto-restart it.

Key definitions

• HEALTHCHECK: Dockerfile instruction that defines the check command, interval, timeout, retries, and start period.
• health status: starting → healthy → unhealthy (three-state model).
• start period: Grace time before Docker begins counting failures — critical for slow-booting containers (e.g., databases, Java apps).

┌──────────────┐      success       ┌──────────────┐
│   starting   │ ▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸ │   healthy    │
└──────────────┘                    └──────────────┘
       │                                 │
       │ failure                        │ failure
       ▼                                 ▼
┌──────────────┐                    ┌──────────────┐
│  unhealthy   │ ◂◂◂◂◂◂◂◂◂◂◂◂◂◂◂◂│   (restart)   │
└──────────────┘                   └──────────────┘

Pro tip: Always set a start_period for apps that need time to initialize — without it, a slow boot counts as repeated failures and flips to unhealthy prematurely.

How it works step by step

The health check lifecycle in Docker:

1. Container starts: Docker immediately sets the status to starting.
2. After start_period expires: Docker runs the HEALTHCHECK command for the first time.
3. Every interval seconds: Docker re-runs the command.
4. If command exits 0 (success): Status becomes healthy.
5. If command exits non-zero: Docker increments a failure counter.
6. When failures reach retries: Status changes to unhealthy.
7. If unhealthy and --restart=always: Docker restarts the container.

The HEALTHCHECK syntax

HEALTHCHECK [OPTIONS] CMD command

Options:
  --interval=DURATION     (default 30s)
  --timeout=DURATION      (default 30s)
  --start-period=DURATION (default 0s)
  --retries=N             (default 3)

Blockquote insight: The health check runs inside the container's namespaces — it has access to localhost and files. Use curl or wget on localhost, not on a public IP.

Hands-on walkthrough

Let's build a real health-checked container from scratch.

Example 1: Basic health check with curl

Create a Dockerfile for a simple HTTP server:

FROM python:3.11-alpine

RUN apk add --no-cache curl

COPY server.py /server.py

HEALTHCHECK --interval=5s --timeout=3s --start-period=10s --retries=2 \
  CMD curl -f http://localhost:8080/health || exit 1

CMD ["python", "-m", "http.server", "8080"]

Build and run:

docker build -t health-check-demo .
docker run -d --name health-demo --restart=always health-check-demo
docker inspect health-demo --format='{{json .State.Health}}' | jq .

Expected output (after 10–20 seconds):

{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2025-01-01T12:00:10Z",
      "End": "2025-01-01T12:00:13Z",
      "ExitCode": 0,
      "Output": ""
    }
  ]
}

Example 2: Simulating a failing health check

Now break the endpoint — remove the /health route (in a real app) or stop the container's internal process. Watch the health status degrade:

docker exec health-demo sh -c "kill -STOP 1"
docker inspect health-demo --format='{{.State.Health.Status}}'
# Output: unhealthy (after retries)

Docker will restart the container because we used --restart=always. Check logs:

docker logs health-demo --tail 5

Example 3: Health check for a database

PostgreSQL health check (no curl needed — check with pg_isready):

FROM postgres:16

HEALTHCHECK --interval=10s --timeout=5s --start-period=30s --retries=5 \
  CMD pg_isready -U postgres || exit 1

Run and verify:

docker build -t pg-health .
docker run -d --name pg-db -e POSTGRES_PASSWORD=secret pg-health
docker inspect pg-db --format='{{.State.Health.Status}}'
# Output: starting → healthy (after ~30 seconds)

Pro tip: For databases, always set a generous start_period (30–60s) because initial startup involves WAL recovery, index creation, and buffer pool warm-up.

Example 4: Custom health check in docker-compose.yml

Health checks can also be defined in Compose (no need to rebuild the image):

version: "3.9"
services:
  web:
    image: nginx:latest
    healthcheck:
      test: ["CMD", "nginx", "-t"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 5s
    restart: always

Start and check status:

docker compose up -d
docker compose ps
# The STATUS column shows "healthy" or "unhealthy"

Compare options / when to choose what

When to choose what: - Use Dockerfile HEALTHCHECK when you own the image and want health logic baked in. - Use Compose healthcheck when you're composing third-party images and can't modify their Dockerfile. - Use external probes (like curl from another container) when you need health from outside the container's network namespace.

Troubleshooting & edge cases

Common health check failures

1. Health check command not found inside container text Error from health check: "executable file not found in $PATH" Fix: Install the tool (e.g., apk add curl) or use a command that exists — like python -c "..." instead of wget.

2. Health check never becomes healthy - Check the log: docker inspect <container> --format='{{json .State.Health.Log}}' | jq .[-1] - Ensure the port/protocol is correct inside the container (use localhost, not 127.0.0.1 for IPv6 configurations). - Increase start_period — especially for Java apps with JVM warmup.

3. Container constantly restarts - If retries is too aggressive (e.g., 1), a transient blip kills the container. - Set retries to at least 3 for typical web apps, 5 for databases.

4. curl hanging forever - The --timeout flag is mandatory in HEALTHCHECK; if omitted, Docker's internal timeout may not kill a hanging curl. - Always use curl -f --max-time 5 http://... to ensure fast failure.

Pro tip: Use docker events to see health status changes in real time: bash docker events --filter 'container=health-demo' --filter 'event=health_status'

What you learned & what's next

You now understand how to manage container health checks — from Dockerfile syntax to Compose configuration, from basic HTTP checks to database probes. You can: - Define HEALTHCHECK instructions for custom images. - Override health checks in docker-compose.yml without rebuilding. - Troubleshoot common health check failures (missing tools, wrong ports, start period misconfiguration). - Monitor container health status using docker inspect and docker events.

Next lesson: Dive into container restart policies — how to pair health checks with restart: always or restart: unless-stopped for true self-healing deployments.

Remember: A running container isn't enough — a healthy container is.