Automate Post-Deployment Smoke Tests

You've just merged a feature, the pipeline turned green, and then the phone rings: the homepage is returning a 500 error in production. Your tests passed in CI, but somehow the real deployment is broken. This is the classic gap between it builds and it works — and it's exactly the pain that automating post-deployment smoke tests solves. By the end of this lesson, you'll know how to catch those failures before your users do, with a lightweight, automated safety net that runs the moment your code hits production.

The Problem This Lesson Solves

Every deployment carries risk, even when every unit test and integration test passes in your CI pipeline. CI validates your code in isolation, but production is a different beast: environment variables might be missing, the database schema could be stale, or a misconfigured load balancer could route traffic to the wrong service. Without a post-deployment check, you only discover these issues when a user reports them — which is too late. Post-deployment smoke tests are your automated "did it actually come up correctly?" check, running immediately after the deploy step completes. They turn the silent, terrifying moment between "deploy succeeded" and "first user hits the site" into a measurable, automated verification.

Core Concept / Mental Model

Think of a smoke test like a pilot's pre-flight checklist: you don't need to test every system fully — just verify that the critical instruments respond, the engines throttle up, and the controls move. If any of those fail, you abort before takeoff. A post-deployment smoke test is the same idea applied to your running application: after your deploy job finishes, you hit the key endpoints, check the database query, and confirm the service responds with the expected status codes. You're not re-running your full test suite — that's what CI already did. You're verifying the runtime environment is healthy, fast, and wired correctly.

Key definitions: - Smoke test: A minimal set of checks to confirm the system is alive and responding — not exhaustive, just indicative. - Post-deployment: The stage in your pipeline after the actual deploy step, when your new version is live. - Automated: Triggered by CI — no human pressing a button, no manual curl commands.

![Mental model diagram in words] Deploy service → Run smoke test → If success: mark release healthy → If failure: roll back or alert. That's the entire loop.

How It Works Step by Step

1. Define the smoke test scope: Pick 3–5 critical endpoints or actions — the homepage, a key API route, a health check, a database-backed query. These should represent the "canary" of your app's health.
2. Write the smoke test script: Use a tool you already know — a Python script with requests, a shell script with curl, or a dedicated tool like smokeshow. The script should assert on HTTP status codes, response times, and maybe key content.
3. Add it to your CI pipeline: In GitHub Actions (or your CI of choice), add a new job that runs after the deploy job completes. Use needs to enforce ordering.
4. Set a timeout and retry logic: Deployments can be momentarily slow to warm up. Add a short retry loop (e.g., 3 attempts with a 10-second pause) to avoid false alarms.
5. Handle failures: If the smoke test fails, the pipeline should fail — and you should trigger an automated rollback or page the on-call engineer. At minimum, record the failure loudly.
6. Monitor the result: Feed the smoke test output into your logging/alerting system (Splunk, DataDog, Slack) so you have a historical record.

Hands-On Walkthrough

Let's build a real, minimal post-deployment smoke test with GitHub Actions and a Python script. We'll assume you have a Flask app deployed to a simple VM endpoint.

Step 1: Write the smoke test script (smoke_test.py):

import sys
import time
import requests

BASE_URL = sys.argv[1] if len(sys.argv) > 1 else "https://your-app.example.com"
ENDPOINTS = [
    "/",
    "/health",
    "/api/status",
]

MAX_RETRIES = 3
RETRY_DELAY = 10  # seconds

def check_endpoint(url, path):
    full_url = url + path
    for attempt in range(1, MAX_RETRIES + 1):
        try:
            resp = requests.get(full_url, timeout=10)
            if resp.status_code == 200:
                print(f"PASS: {path} -> 200")
                return True
            else:
                print(f"FAIL (attempt {attempt}): {path} -> {resp.status_code}")
        except requests.exceptions.RequestException as e:
            print(f"ERROR (attempt {attempt}): {path} -> {e}")
        if attempt < MAX_RETRIES:
            time.sleep(RETRY_DELAY)
    return False

all_passed = True
for endpoint in ENDPOINTS:
    if not check_endpoint(BASE_URL, endpoint):
        all_passed = False

sys.exit(0 if all_passed else 1)

Step 2: Add the CI job (.github/workflows/deploy.yml):

name: Deploy and Smoke Test

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        run: |
          echo "Deploying your app..."
          # Your real deploy script goes here
          # e.g., ssh user@server 'systemctl restart myapp'
          sleep 5  # simulate deploy time
          echo "Deploy complete"

  smoke-test:
    needs: deploy
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Run smoke tests
        env:
          APP_URL: ${{ secrets.APP_URL }}
        run: |
          python smoke_test.py $APP_URL

Step 3: Expected output when you push a commit:

PASS: / -> 200
PASS: /health -> 200
PASS: /api/status -> 200
All smoke tests passed.

If the service is down, you'll see:

FAIL (attempt 1): / -> 500
FAIL (attempt 2): / -> 500
FAIL (attempt 3): / -> 500

And the pipeline will fail, preventing you from thinking the release is healthy when it isn't.

Compare Options / When to Choose What

You don't have to write a custom script every time. Here's a comparison of common approaches:

When to choose what: - For a small service with one endpoint, a simple curl in your workflow is fine. - For a microservices architecture with several endpoints and retry needs, a custom Python script gives you control. - For mission-critical production, consider a dedicated solution like checkly that also gives you continuous monitoring after deployment, not just during it.

Troubleshooting & Edge Cases

1. Smoke test passes, but the app is still broken. Your checks are too shallow. Expand the test to verify data — check that a specific JSON field exists, or that the database returns the expected count. Fix: Add content assertions, not just status code.

2. The test fails immediately after deploy due to warm-up time. You see a panic in the team. Fix: Use retries with a delay — we already did this in our script. Increase RETRY_DELAY to 30–60 seconds for slow-starting apps.

3. Flaky test due to network hiccups between CI and your server. A transient error causes false positives. Fix: Add retries (done), and also consider running the smoke test from the same network as your users (or a separate staging VM) to reduce network variability.

4. Secrets are not available in the smoke test job. You get a KeyError or empty URL. Fix: Add the secrets to the job explicitly, as we did with APP_URL in the example.

5. The smoke test passes but takes too long, slowing your pipeline. Users complain. Fix: Run smoke tests in parallel with other post-deploy jobs if there are no dependencies, or limit the number of endpoints to the most critical ones.

What You Learned & What's Next

You now understand the gap between "deploy works" and "app actually works" — and you have a practical, automated way to bridge it. You've learned: - The core idea of automate post-deployment smoke tests: a minimal, automated check on the live system. - How to write a simple Python smoke test with retries. - How to plug it into a GitHub Actions workflow using needs. - How to compare alternative tools and when to pick each. - How to troubleshoot common edge cases like warm-up and flaky networks.

This is a huge stepping stone for your CI/CD foundations. Next, you'll explore automated rollback strategies — what to do when a smoke test fails, and how to revert to the last known-good version without panic. That's the natural continuation: smoke tests tell you when something's wrong; rollback tells you how to recover.

But before you move on, take the knowledge you've gained and apply it — because the best way to lock it in is to build it yourself.