Cache Dependencies for Faster Builds

CI/CD pipelines are designed to deliver fast feedback, yet every commit can feel like a slow boat to deployment when npm install, pip install, or bundle install re-downloads the same packages over and over. Nothing wastes more developer time than a five-minute dependency download on a dependency that hasn't changed in weeks. Caching dependencies is the CI/CD superpower that turns those five-minute waits into seconds — and in this lesson, you'll learn exactly how to do it for your own pipelines.

The problem this lesson solves

Almost every CI/CD pipeline has at least one step that downloads external dependencies: JavaScript needs node_modules, Python needs site-packages, Ruby needs gems. By default, most CI systems treat every build as a clean slate — a fresh virtual machine or container with nothing pre-installed. That means your pipeline downloads the same packages, thousands of times, on every single commit.

Let's put real numbers on the pain. Consider a typical Node.js project installed from scratch:

Downloading dependencies: 142s
Installing dependencies: 58s
Running tests: 12s
Total build time: 212s

The dependency phase alone eats up 94% of that build's execution time. Now multiply that by the number of commits, pull requests, and pushes your team makes each day. You’re not just wasting compute; you’re slowing down developer feedback and burning through your CI minutes budget. Caching attacks this problem directly by persisting the useful parts of a build between runs.

Pro tip: If you check your CI provider's billing dashboard, you'll often find that dependency downloads are the single biggest contributor to build time. Cutting those downloads is the highest-ROI optimization available.

The problem is also about consistency and reliability. Every time you re-download a package, you risk a network hiccup, a transient registry outage, or a package that was yanked from the repository. Caching injects a layer of stability, because you're typically reusing packages your build already verified.

Core concept / mental model

The mental model for caching is simple: your CI pipeline is a short-lived worker that can remember work from previous shifts.

Think of it like a chef who prepares a mise en place every morning. If the chef threw away all their sliced vegetables and spices every night, they'd spend half the morning re-slicing the same onions. Instead, they store the prepped materials in labeled containers and reuse them the next day — that's a cached dependency.

The core unit in most caching systems is a cache key. A cache key is a string that identifies a unique version of your dependency set. It's usually generated from a combination of:

• The dependency manifest file (e.g., package-lock.json, Pipfile.lock, Gemfile.lock)
• The build environment (OS, Python version, Node version)
• Sometimes a date, to allow a periodic refresh

When your pipeline starts, it presents its cache key to the cache service. If the service has a cache entry matching that key, it restores the cached dependencies. If not, your pipeline downloads from scratch and, at the end, saves the new cache for the next run.

Definitions you need to know:

• Cache hit: When your cache key matches an existing cache entry, and dependencies are restored from cache.
• Cache miss: When no matching cache entry exists, and you install from scratch.
• Cache poisoning: When bad data gets into your cache and gets reused — avoid by careful key design.
• Cache invalidation: The process of intentionally making old caches unusable (e.g., when you bump a dependency or change a lockfile).

This mental model applies across all major CI platforms — GitHub Actions, GitLab CI, CircleCI, and Jenkins. They differ in syntax but share the same underlying idea: save by a key, restore by a key, invalidate by a careful key choice.

How it works step by step

The general flow of caching in a CI pipeline is almost always the same. Here’s the high-level sequence:

1. Compute a cache key from the dependency manifest (and maybe the toolchain version).
2. Check for a cache entry using that key. - If found, restore the dependencies into the expected directory (e.g., node_modules). - If not found, skip restoration.
3. Run your installation step (e.g., npm ci, pip install -r requirements.txt). - This validates the cache or creates the dependencies from scratch.
4. After a successful install, save the dependencies to the cache under the same key (or a fallback key).
5. Run your tests (or build) — now they execute against the fast installed dependencies.

The key thing to remember is order matters. You must restore before you build, and you must save after the install (but usually before or after tests, as long as it's on a successful run).

In GitHub Actions, the actions/cache action is the standard tool. Here’s a practical example for a Node.js project:

name: CI
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Set up Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'  # This turns on built-in caching for npm

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

In the example above, setup-node automatically caches the npm cache directory. But for more control, use actions/cache explicitly:

- name: Cache node_modules
  uses: actions/cache@v4
  with:
    path: node_modules
    key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

The key uses hashFiles('package-lock.json'), which creates a unique hash of your lockfile. When that file changes, the key changes, and the old cache is discarded — exactly what you want.

The restore-keys list is a fallback: if an exact match isn't found, the pipeline will use the most recent cache that matches the prefix. This is useful when your lockfile changes but you still want to reuse most of the old dependencies.

Pro tip: Always use a lockfile (package-lock.json, Pipfile.lock, Gemfile.lock) for accurate cache keys. If you hash only package.json, a cache hit may use dependencies that no longer match your intended versions.

Hands-on walkthrough

Let's build a complete, practical pipeline that caches Python dependencies with pip. We'll use GitHub Actions, but the pattern transfers to any CI.

Step 1: Create a minimal Python project

Create a simple project structure:

my-project/
├── requirements.txt
├── app.py
└── .github/workflows/ci.yml

requirements.txt:

requests==2.31.0
flask==3.0.0
pytest==7.4.0

app.py:

def hello():
    return "Hello, world!"

if __name__ == "__main__":
    print(hello())

Step 2: Write the CI workflow with caching

.github/workflows/ci.yml:

name: CI with Caching
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Cache pip
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest

Step 3: Run the pipeline and observe the cache behavior

First run (cache miss) output:

Cache miss: key=Linux-pip-8f3d2c1f...
Installing dependencies...
Successfully installed requests-2.31.0 flask-3.0.0 pytest-7.4.0
Post job: Saving cache with key Linux-pip-8f3d2c1f...

Second run (no changes) output:

Cache hit: key=Linux-pip-8f3d2c1f...
Installing dependencies (using cache)...
Tests complete in 4 seconds (vs 25s before).

Now edit requirements.txt to add httpx==0.27.0, commit, and rerun:

Cache miss: key=Linux-pip-9f4e8a1b...
Installing dependencies (some new packages)...

The key changed, so the cache was invalidated — and a fresh cache was saved at the end.

Step 4: Verify with a script

Run a local test to confirm the workflow logic:

# cache_demo.py
import hashlib

def compute_cache_key(lockfile_path):
    with open(lockfile_path, 'rb') as f:
        content = f.read()
    return hashlib.sha256(content).hexdigest()

key1 = compute_cache_key('requirements.txt')
print(f"Cache key: {key1}")
# If you change requirements.txt and rerun, key will differ.

When you run it, you'll see a hex hash — that's your cache key.

Compare options / when to choose what

Not all caching approaches are equal. The table below compares the most common methods for dependency caching:

When to choose what:

• Start with package-manager cache — it’s the simplest and works everywhere.
• If your install step is still slow, move to caching the dependency directory (but always base the key on the lockfile).
• Use Docker image caching when you already have a container-based pipeline and you want to avoid install steps entirely.
• Vendoring is a rare choice, but perfect for air-gapped environments.

Pro tip: In GitHub Actions, actions/setup-node and actions/setup-python now support a cache: 'npm' (or 'pip') option that automatically uses the best-practice cache path for you. Use it when possible to avoid manual key management.

Troubleshooting & edge cases

Caching is powerful, but it introduces new failure modes. Here are the most common issues and how to fix them:

1. Cache hit but dependencies are missing or incomplete

Symptom: Your pipeline restores a cache, but npm ci still fails because node_modules is partially missing.

Cause: You cached the wrong directory (e.g., ~/.npm instead of the actual install), or you used a cache key that doesn't change when the lockfile does.

Fix: Double-check the cache path. For npm, the package-manager cache is ~/.npm, but the installed packages are in node_modules. If you cache node_modules, ensure your key includes hashFiles('package-lock.json').

2. Stale cache after a dependency update

Symptom: You bump a package version, but the build still uses the old one.

Cause: The cache key didn't change — perhaps you forgot to update the lockfile, or you're hashing the wrong file.

Fix: Always generate the key from a lockfile, not a manifest. For example, use package-lock.json instead of package.json. If you don't have a lockfile, consider generating one.

3. Cache exceeds size limits

Symptom: Your CI provider errors with "cache size limit exceeded."

Cause: You're caching huge directories like the entire ~/.cache or node_modules for multiple versions.

Fix: Split caches by OS and toolchain version, or cache only the essential directory. In GitHub Actions, you can use multiple actions/cache steps with distinct keys for different parts.

4. Cache key changes on every run (cache never hits)

Symptom: The hashFiles output seems different every time, even with no changes.

Cause: You're hashing a file that changes due to metadata, like package.json timestamps, or you're including a timestamp in the key.

Fix: Use only content-based hashing (e.g., hashFiles on lockfiles). Avoid including current date in the main key — use it only as a fallback in restore-keys to allow periodic refresh.

5. Cache restoration is slower than a fresh install

Symptom: The build is slower with caching enabled than without.

Cause: You're caching too much, or the cache path is on a slow network volume. On GitHub Actions, restore speeds are usually fast, but on self-hosted runners, disk I/O can bottleneck.

Fix: Test different cache paths, and ensure you only cache dependencies, not build artifacts that change every time. Use actions/cache's lookup-only option if you only want to restore and never save (for read-only jobs).

Pro tip: When debugging, use the --verbose flag on actions/cache or check the step logs to see the exact cache keys and hit/miss status. Most providers log this clearly.

What you learned & what's next

Let's recap what you've mastered in this lesson:

• You can explain the core idea behind caching dependencies: persist the expensive part of a build between runs so every commit doesn't re-download the world.
• You can implement caching in a practical exercise: you built a GitHub Actions workflow that caches pip dependencies using a lockfile-based cache key, and you saw how hits and misses behave.
• You know when to choose different caching strategies: package-manager cache vs. dependency directory vs. Docker images, and you can weigh the trade-offs.
• You can troubleshoot common caching pitfalls: stale keys, oversized caches, and path mistakes.

Cache dependencies for faster builds is a foundational skill that pays off on every single pipeline you ever write. It's the difference between a CI system that feels painfully slow and one that feels instant.

Your next lesson in this CI/CD foundations track is "Deploy previews / environments" (coming next). You'll take your fast, cached pipeline and add the ability to deploy ephemeral previews for every pull request — a perfect pairing with the reliability you've just gained.

Keep this lesson close: as you move into previews and environment-based workflows, you'll re-use these caching patterns to keep every deploy instant. Master this, and you're well on your way to CI/CD mastery.