Run Python in CI Pipelines

Every DevOps engineer has felt the sting: a script that works perfectly on your laptop fails in CI. The Python environment is different, dependencies are missing, or the working directory isn't what you expected. You spend hours debugging a pipeline that should take minutes. This lesson solves that pain by teaching you how to run Python in CI pipelines reliably and reproducibly, so your automation runs the same everywhere.

The problem this lesson solves

Running Python in CI pipelines is about reproducibility. When you run python script.py locally, you're using your machine's environment. In CI, you're starting from a clean slate. Without a deliberate strategy, you'll face:

• Missing dependencies — ModuleNotFoundError because pip install wasn't run.
• Different Python versions — code that uses 3.10+ features fails on a 3.8 base image.
• Environment variables — secrets and configuration that exist locally but not in CI.
• Working directory issues — relative paths that break because CI checks out code to a different folder.
• Caching problems — every build reinstalls everything, slowing the pipeline.

The core problem is that your script and its environment are not the same thing — and CI forces you to be explicit about both.

Core concept / mental model

Think of a CI pipeline as a factory assembly line. Each job is a station that performs a specific task. To run Python in CI pipelines, you need to prepare the station before the worker (your script) arrives.

The mental model has three layers:

1. The Python interpreter — the runtime that executes your code.
2. The dependency environment — packages your script imports (requests, boto3, etc.).
3. The execution context — working directory, environment variables, and system tools.

A good CI setup treats these layers as infrastructure as code: you declare exactly what you need, and the pipeline provides it. This is why tools like requirements.txt, venv, and docker exist — they encode the environment.

Pro tip: Treat your CI pipeline as a stateless runner — each job starts fresh, runs your script, and then discards everything. This forces you to make your scripts explicit and reproducible.

graph LR
    A[Code push] --> B[CI Trigger]
    B --> C[Checkout repo]
    C --> D[Setup Python]
    D --> E[Install deps]
    E --> F[Run tests/script]
    F --> G[Report results]

How it works step by step

Every CI pipeline that runs Python follows the same logical sequence. Whether you use GitHub Actions, GitLab CI, or Jenkins, the steps are:

1. Trigger the pipeline — typically on a push, pull request, or schedule.
2. Check out the code — pull the repository to the runner.
3. Set up Python — select the version and install the interpreter.
4. Create an isolated environment — use venv or a container to avoid conflicts.
5. Install dependencies — read requirements.txt or pyproject.toml and install.
6. Run your scripts or tests — execute the Python command that drives your automation.
7. Collect results and cache — log output, upload artifacts, and cache dependencies for the next run.

Let's break down each step with a typical GitHub Actions example.

Hands-on walkthrough

Let's build a realistic CI pipeline that runs a Python script for a DevOps task — say, checking that all AWS regions have a particular tag.

Step 1: Create a simple Python script

Create check_tags.py that uses boto3 to verify a tag on all EC2 instances:

import boto3

def check_tags():
    ec2 = boto3.client('ec2', region_name='us-east-1')
    response = ec2.describe_instances()
    missing = []
    for reservation in response['Reservations']:
        for instance in reservation['Instances']:
            tags = {t['Key'] for t in instance.get('Tags', [])}
            if 'Team' not in tags:
                missing.append(instance['InstanceId'])
    if missing:
        raise RuntimeError(f"Instances missing Team tag: {missing}")
    print("All instances tagged correctly.")

if __name__ == "__main__":
    check_tags()

Step 2: Create a requirements.txt file

boto3==1.34.145

Step 3: Write a GitHub Actions workflow

Create .github/workflows/ci.yml:

name: Python CI
on: [push, pull_request]

jobs:
  run-script:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Create virtual environment
        run: |
          python -m venv .venv
          source .venv/bin/activate

      - name: Install dependencies
        run: |
          pip install -r requirements.txt

      - name: Run script
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          python check_tags.py

Expected output in the CI log:

All instances tagged correctly.

Step 4: Add caching for speed

To avoid reinstalling dependencies on every run, add a cache:

- name: Cache pip
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
    restore-keys: |
      ${{ runner.os }}-pip-

With this, run Python in CI pipelines becomes faster and more efficient.

Compare options / when to choose what

There are multiple ways to run Python in CI pipelines. Here's a comparison:

Recommendations:

• For most DevOps scripts, use setup-python with venv — it's the simplest and works everywhere.
• If your script needs system libraries (e.g., libpq for PostgreSQL), use a Docker image.
• If you need binary packages like numpy, use Conda or a pre-built wheel.

Pro tip: Always pin your Python version and dependency versions. Unpinned versions are a classic source of CI flakiness.

Troubleshooting & edge cases

ModuleNotFoundError

If the script can't find a module, it usually means the virtual environment wasn't activated or dependencies weren't installed. Check your pipeline logs for the pip install step, and make sure you're using the same Python interpreter.

python: command not found

On some runners, python points to Python 2. Use python3 explicitly or set up the interpreter properly with actions/setup-python.

Path issues

If your script uses relative paths, CI's working directory is usually the repo root. Use os.path.dirname(__file__) to be safe:

from pathlib import Path
BASE_DIR = Path(__file__).resolve().parent

Secrets not available

If your script needs AWS credentials, ensure they are set as environment variables in the pipeline. For GitHub Actions, use secrets and pass them as env.

Cache misses

If you change requirements.txt often, the cache will miss frequently. That's fine — it's better to be correct than fast.

What you learned & what's next

You now know how to run Python in CI pipelines: set up the interpreter, create a virtual environment, install dependencies, run scripts, and handle common pitfalls. You have a reusable pattern: checkout → setup → venv → pip install → run.

The next lesson in this track will cover integrating Python with code quality tools — using linters and formatters in CI to catch issues before they reach production.

Now go ahead and make your CI pipelines as reproducible as your local environment!