The Day Our CI Pipeline Almost Cost Us Everything

The Day Our CI Pipeline Almost Cost Us Everything

It started with a single package update. One dependency bump in a pull request. The tests passed. The build succeeded. It looked completely harmless. Except it wasn't.

That package was a typosquat — requets instead of requests. Someone had pushed it to PyPI, and our automated CI/CD pipeline happily pulled it in. We caught it during a manual code review, but barely. That near-miss sparked a complete rethink of how we handle our software supply chain.

Why Your Pipeline Is a Prime Target

Modern CI/CD pipelines are automated dependency vacuums. They pull from public registries, run untrusted code, and deploy with minimal oversight. Each integration point is a potential entry vector.

The 2023 Sonatype report found that over 245,000 malicious packages were discovered in public repositories — a 742% increase over the previous two years. And those are just the ones we know about.

The Three Pillars of Supply Chain Security

1. Pin Everything, But Pin Smart

The classic approach — freeze your dependencies with exact version numbers — has a known weakness: you miss critical security patches. The smarter approach is a graduated pinning strategy:

# requirements.txt — for production builds only
certifi==2023.11.17
requests==2.31.0
cryptography==41.0.7

But for development, pin to major.minor ranges:

# dev-requirements.txt
certifi~=2023.11
requests~=2.31

Then use automated tools like Dependabot or Renovate to create PRs for minor updates. Each update triggers a fresh review cycle.

2. Hash Verification — Your First Line of Defense

Pin a version, and someone can still replace the package on the registry. Hash verification protects against that.

# Dockerfile example
RUN pip install --require-hashes -r requirements.txt

This forces pip to check SHA-256 hashes you've pre-computed. Generate them with:

pip freeze --require-hashes > requirements.txt

Store these in a separate, signed file. In your CI pipeline, validate the signature before allowing the build.

3. SBOM — The Accounting Ledger for Your Code

A Software Bill of Materials (SBOM) is a machine-readable inventory of every component in your build. Think of it as the ingredient label for your application.

Generate one automatically in your CI pipeline:

# GitHub Actions example
- name: Generate SBOM
  run: |
    cyclonedx-py requirements.txt
    cat cyclonedx-report.json
- name: Upload SBOM
  uses: actions/upload-artifact@v4
  with:
    name: sbom
    path: cyclonedx-report.json

Store every SBOM in a versioned repository. When a vulnerability is announced, you can instantly answer: "Are we affected?"

Practical Pipeline Hardening Steps

Step 1: Isolate the Build Environment Never run builds on shared runners with internet access. Use private runners with controlled egress:

jobs:
  build:
    runs-on: [self-hosted, secure-build]
    steps:
      - uses: actions/checkout@v4
      - name: Build with restricted network
        run: |
          # Only allow connections to private registry mirrors
          DOCKER_BUILDKIT=1 docker build --network=host

Step 2: Vet the Vetters Your automated security scanners (Snyk, Trivy, etc.) are themselves dependencies. Pin their versions and verify their signatures:

- name: Install Trivy
  run: |
    curl -L -o trivy.tar.gz https://github.com/aquasecurity/trivy/releases/download/v0.47.0/trivy_0.47.0_linux_amd64.tar.gz
    echo "expected-sha256-of-trivy-tar.gz trivy.tar.gz" | sha256sum -c -
    tar -xzf trivy.tar.gz

Step 3: Enforce Multi-Party Approval No single developer should be able to push a dependency update to production. Use branch protection rules that require: - Two code reviews for any requirements.txt or package.json change - A separate security team member's approval for new dependencies - A 24-hour hold period on dependency updates to detect suspicious patterns

The Hard Truth About Automation

You can't automate your way out of supply chain risk. Automated scanners catch known vulnerabilities — they won't stop a backdoor injected by a trusted maintainer who had their account compromised.

The real defense is a combination of automation and human judgment. Build the pipeline to block anything that doesn't pass your checks, but give human reviewers the tools and time to evaluate what's truly new:

# Custom pre-commit hook example
import requests
import hashlib

def check_new_dependency(package_name, version):
    # Query VirusTotal or similar API for package intelligence
    response = requests.get(
        f"https://api.example.com/packages/{package_name}/{version}/risk"
    )
    risk_score = response.json()["risk_score"]
    if risk_score > 0.7:
        print(f"WARNING: {package_name}@{version} has high risk score")
        exit(1)

What's Next in Supply Chain Security

The industry is moving toward signed provenance — cryptographic attestations that prove exactly how a package was built. GitHub's Sigstore and the in-toto framework are early examples.

For now, the lowest-hanging fruit is often the most effective: stop trusting automatically. Treat every dependency update as a potential attack, because in the current landscape, it might be.

The package that almost slipped into our pipeline wasn't sophisticated. It was a simple spelling trick that our automation never questioned. Fixing the automation was easy. Changing the culture — from "trust the build" to "question everything" — was slower, but that's what actually kept us safe.