Run Tests in CI

Your tests pass locally, so why does the pipeline keep failing? You could be chasing a phantom, or you could be missing the point: running tests in a CI pipeline isn't just about re-running what you do locally — it's about creating a reproducible, automated gate that runs your entire test suite on every commit, in a clean environment, before the code ever reaches main. This lesson walks you through the why, the how, and the troubleshooting of wiring tests into a GitHub Actions pipeline, so you can stop worrying about "works on my machine" and start trusting your commits.

The problem this lesson solves

You've just pushed a feature branch. You ran pytest locally, everything is green, and you're proud of your work. But when the pull request lands, the CI bot shows a red ✗. Panic sets in. You dig through logs, re-run tests, and discover that a hidden dependency wasn't installed, a Python version mismatch slipped through, or you forgot to commit a fixture file.

Without a CI pipeline running your tests, every developer is a silent island. Each person's machine has its own quirks: stale virtual environments, different OS versions, forgotten environment variables. The problem isn't your tests — it's that nobody ever ran them in a controlled, consistent environment. Running tests in a CI pipeline turns that chaotic process into a repeatable gate that runs the same test commands, on the same Python version, with the same dependencies, every single time.

By the end of this lesson, you'll be able to explain the core idea behind running tests in CI and complete a hands-on exercise that gets you from a local test suite to a green pipeline badge on your repo.

Core concept / mental model

Think of your CI pipeline as a conveyor belt in a factory. Each commit enters the belt, passes through a series of stations: linting, unit tests, integration tests, packaging, artifact storage. Your job at this stage of the belt is the test station — the place where the product is checked for defects before it's allowed to move forward.

The heart of the model is the test job. A job in GitHub Actions is a group of steps that run on a fresh runner (an ephemeral virtual machine or container). Each job has a runner environment — typically an Ubuntu image with Python pre-installed. The test job does three things in sequence:

1. Check out your code — bring the repository contents onto the runner.
2. Set up the test environment — install Python, install dependencies from requirements.txt or pyproject.toml.
3. Execute the test commands — run your test runner (pytest, unittest, etc.) and capture the exit code.

A pipeline, in the CI/CD sense, is the full workflow definition — in GitHub Actions, a YAML file in .github/workflows/ that lists jobs and their triggers. The pipeline is the skeleton; the test job is the muscle that executes your tests.

Mental model checkpoint: If your tests fail locally, they'll fail in CI. But if they pass locally and fail in CI, the environment differs — that's exactly why the pipeline is valuable: it catches those differences automatically.

How it works step by step

Running tests in a CI pipeline is a sequence of events you can reason about from first principles.

1. Trigger: The pipeline is activated by an event — a push to main, a pull request, or even a manual workflow dispatch.
2. Runner spawn: GitHub Actions provisions a fresh runner (a virtual machine) with the OS you specify (e.g., ubuntu-latest).
3. Checkout action: The actions/checkout step downloads your repository's code onto the runner.
4. Environment setup: You use actions/setup-python to pin a Python version — this is crucial because it makes your test environment deterministic.
5. Dependency installation: You install the packages your project needs, ideally with pinned versions from a lock file. This step is your main defense against "works on my machine".
6. Test execution: You run your test command, such as pytest -q or python -m unittest. The command's exit code determines the job's fate — a non-zero exit code fails the pipeline.
7. Reporting: The pipeline surface (UI, PR comments) shows pass/fail status, and logs are stored for debugging.

The beauty is the cause-effect chain: a clean environment + explicit dependencies + explicit test commands = a deterministic test run. If the tests pass in CI, you have high confidence they'll pass for anyone else who pulls your code.

Hands-on walkthrough

Let's turn that mental model into a working pipeline. We'll use GitHub Actions and a minimal Python project with a couple of tests.

1. Create a minimal test suite

First, create a Python file with a function and a test file:

# calculator.py
def add(a, b):
    """Add two numbers."""
    return a + b


def divide(a, b):
    """Divide two numbers, raising on zero divisor."""
    if b == 0:
        raise ValueError("Cannot divide by zero")
    return a / b

# test_calculator.py
import pytest
from calculator import add, divide


def test_add():
    assert add(2, 3) == 5


def test_divide():
    assert divide(10, 2) == 5


def test_divide_by_zero():
    with pytest.raises(ValueError):
        divide(10, 0)

2. Write the CI workflow

Create .github/workflows/tests.yml in your repository:

name: Run tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install pytest

      - name: Run tests
        run: pytest -q

Expected output (in the "Run tests" step log):

============================= test session starts =============================
collected 3 items

test_calculator.py ...                                                [100%]

============================== 3 passed in 0.15s ==============================

The pytest -q command's exit code is 0, so the „Run tests" step (and the job) succeeds. If any test fails, pytest exits with a non-zero code, marking the job as failed.

3. Add a bonus step: fast but safe

To make your pipeline even more practical, add a step to fail fast on syntax errors with py_compile (or use --strict flags in pytest):

      - name: Compile all Python files
        run: |
          python -m compileall .

      - name: Run tests
        run: pytest -q

Now your pipeline doesn't just run tests — it does a quick syntax check first, catching errors before the slower test run kicks in.

4. Run it and see it work

Push your code and open a pull request (or just push to main). Watch the Actions tab. You'll see the job „test" appear, and each step expands to show logs. When the pipeline goes green, you've successfully run tests in a CI pipeline.

Pro tip: Name your steps clearly (e.g., "Run unit tests"). When a step fails, the step name instantly tells you what went wrong, saving you minutes of log-diving.

Compare options / when to choose what

You don't have to use pytest specifically. Here's a comparison of common testing setups you might encounter in a CI pipeline:

For most projects, pytest is the sweet spot: it's readable, extensible, and the de-facto standard in modern Python. unittest is a zero-dependency option if you want to keep your pipeline lean. tox shines when you need to test against several Python versions in the same job — but for CI simplicity, you often run multiple jobs with different python-version values instead.

When to choose what: If you're working on a library, run tests across versions with a matrix (see variations). If you're building an internal app, one Python version with pytest is usually enough.

Rule of thumb: The test command you use locally should be exactly the one in your pipeline. If you'd type pytest in your terminal, your pipeline should call pytest, not run_tests.sh that does something else.

Troubleshooting & edge cases

Even the best pipelines hit snags. Here are the most common issues you'll face when running tests in CI, and how to fix them.

1. Tests pass locally, fail in CI

Symptom: Everything is green on your machine, but the pipeline fails.

Possible causes & fixes: - Missing dependency — your requirements.txt is incomplete. Compare pip freeze locally to what's in your CI environment. Add the missing package to your requirements file. - Python version mismatch — you're on Python 3.11, CI uses 3.12. Pin the exact version in actions/setup-python or use a matrix. - Environment variables — your tests rely on an env var set in your .env file, but CI doesn't have it. Set them as secrets or in the workflow's env: block. - File path issues — your tests create files relative to a hardcoded path. Use Path(__file__) or os.path.dirname(__file__) to make paths portable.

2. "ModuleNotFoundError: No module named 'pytest'"

You installed dependencies, but pytest is missing. Fix: include pytest in your requirements file (or install it explicitly before running tests). Even better, use a requirements-dev.txt that includes both runtime and dev dependencies.

3. Pipeline passes but tests actually didn't run

Your pipeline is green, but you never see test output. This happens when your test command is wrong, e.g., pytest finds no tests because your test files don't match the default pattern. Fix: use pytest -q test/ or create a pytest.ini file. Always check the logs — the test runner should report "3 passed" or similar.

4. Git submodules not checked out

If your repo has submodules, actions/checkout@v4 only checks out the top-level repo. Add submodules: recursive to the checkout step.

5. Rate limits or flaky network

Sometimes installing dependencies times out. Retry logic isn't built-in by default, but you can use pip install --retries 3 or split installation into a separate step to cache results better.

Debugging tip: If a step fails, click on the step in the Actions tab, then expand the log. The last 10 lines usually tell you the real error — the rest is noise.

What you learned & what's next

You now understand the core idea behind running tests in a CI pipeline: it's a deterministic, automated gate that catches environment differences before they become production bugs. You've completed a hands-on exercise that sets up a GitHub Actions workflow, runs pytest, and reports a green or red status. You can explain the core idea and apply it in practice.

What's next: In the next lesson of the CI/CD foundations track, you'll learn how to cache dependencies so your pipeline runs faster, and then how to upload test artifacts (like coverage reports) for later inspection. These build naturally on the test job you just created.

Go ahead and add a cache step to your pipeline from this lesson — try it, break it, and fix it. That's how you cement the knowledge. You're on your way to trusting your pipeline, not just your laptop.