AWS CodePipeline CI/CD Pipeline

You’ve just pushed a new commit to your repository, and now you’re waiting for someone to log into a server, run git pull, restart the service, and hope the tests pass. That workflow is slow, error-prone, and doesn’t scale beyond one project. Build a CI/CD pipeline with AWS CodePipeline to automate the path from commit to deployment. In this lesson, you’ll learn how to orchestrate builds, tests, and deployments using AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy, and you’ll finish with a working pipeline for a Python application. This isn’t just a tool walkthrough—it’s the shift from “it works on my machine” to “it works in production, every time.”

The problem this lesson solves

Manual deployment workflows are the single biggest source of production outages and developer burnout. When you deploy by hand, you risk:

• Configuration drift — the server that was set up months ago no longer matches your latest code.
• Untested changes — the code that passes on your laptop fails in the cloud because of missing dependencies or environment differences.
• Slow feedback — bugs are discovered hours or days after a commit, not in the minutes after you push.

A CI/CD pipeline solves these by automating every step from code push to production. Continuous Integration (CI) means every commit triggers a build and test suite. Continuous Delivery (CD) means the tested artifact is automatically deployed to staging, and with the flip of a switch, to production. AWS CodePipeline is the AWS-native service that orchestrates this whole flow. It’s not just for web apps—it works for Python packages, serverless functions, and even machine learning models.

By the end of this lesson, you’ll be able to explain why “deploy on every commit” is a team superpower, and you’ll have the skills to build that pipeline yourself.

Core concept / mental model

Think of AWS CodePipeline as the assembly line for your software. Each stage is a workstation: source code is pulled in, tests run, artifacts are built, and finally the package is shipped to a deployment target. CodePipeline orchestrates the sequence, passing the output of one stage as the input to the next.

The three core services

• AWS CodePipeline — the orchestrator. It defines stages and transitions, triggering the next stage only when the previous one succeeds.
• AWS CodeBuild — the worker. It runs your build commands (e.g., pip install -r requirements.txt, pytest) in a managed, disposable environment. It produces a build artifact—a ZIP or Docker image, for example—that downstream stages use.
• AWS CodeDeploy — the deployer. It takes the artifact and deploys it to a fleet of EC2 instances, an Auto Scaling group, or even an on-premises server.

But here’s the key: CodePipeline is not limited to AWS-native tools. You can plug in GitHub or Bitbucket as the source, Jenkins as the build provider, or even a Lambda function as a custom action. The mental model is always the same: a directed graph of stages that your code flows through.

What is a “pipeline” in this context?

A pipeline is a series of stages—for example, Source, Build, Deploy. Each stage contains one or more actions. Actions are atomic tasks like “pull from GitHub” or “run CodeBuild project.” The pipeline runs your code through these stages in order, stopping and failing loudly if anything goes wrong.

Here’s how a typical pipeline looks for a Python service:

That’s the core idea: every push to your repository triggers a fully automated, repeatable path to production.

How it works step by step

The magic happens through state transitions. Let’s walk through what happens after you push a commit to GitHub:

1. Source stage triggers. CodePipeline uses a webhook (or polling) to detect the new commit on the configured branch. It fetches the source code and passes it to the next stage.
2. Build stage runs. CodeBuild starts a fresh container. It reads your buildspec.yml file, which tells it exactly what commands to run (install dependencies, run tests, package the app). On success, it produces an artifact and stores it in an S3 bucket.
3. Deployment stage executes. CodeDeploy receives the artifact, copies it to your EC2 instances, and runs the lifecycle scripts (stop the old app, install the new one, start it again). It performs health checks and rolls back automatically if the deployment fails.
4. Pipeline completes. If all stages succeed, the pipeline goes to the Succeeded state. If any stage fails, the pipeline stops, and you get alerts (via CloudWatch Events, for example).

The beauty is that you get visibility — the CodePipeline console shows a visual graph with green (success) or red (failure) at each stage. You can inspect logs for each action without SSHing into a server.

The buildspec.yml file

The heart of the build stage is buildspec.yml. It tells CodeBuild what to do. A minimal Python example looks like this:

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.11
    commands:
      - pip install --upgrade pip
      - pip install -r requirements.txt
  build:
    commands:
      - pytest
      - python -m compileall .
artifacts:
  files:
    - '**/*'

As soon as the build command exits with a non-zero exit code, CodeBuild marks the build as failed. The pipeline stops. That’s the fail-fast philosophy.

Hands-on walkthrough

Let’s build a complete pipeline for a simple Python Flask app using the AWS console and CLI. We’ll assume you have an AWS account, the AWS CLI installed, and a GitHub repository with your code. This walkthrough mirrors what you’d do for any Python web service.

Step 1: Prepare your application

Create a minimal Flask app and commit it to a GitHub repository. Here’s an example structure:

# app.py
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello():
    return "Hello, CI/CD!"

Also add a simple test:

# test_app.py
from app import app

def test_hello():
    client = app.test_client()
    resp = client.get('/')
    assert resp.data == b"Hello, CI/CD!"

Add requirements.txt with flask and pytest. Commit and push to GitHub.

Step 2: Create an S3 bucket for artifacts

CodePipeline stores build artifacts in S3. Create a bucket (if you don’t have one):

aws s3 mb s3://my-codepipeline-artifacts

Step 3: Create a CodeBuild project

In the CodeBuild console, Create build project. Give it a name like my-python-build. Choose your source provider (GitHub) and connect to your repo. For the buildspec, choose Use a buildspec file and point to buildspec.yml in your repository root. Leave the rest as defaults (Ubuntu, standard image).

Verify the project works by running Start build and checking that the logs show pytest passing.

Step 4: Create a CodePipeline

In the CodePipeline console, Create pipeline. Name it my-python-pipeline. Add stages:

1. Source — choose GitHub, select your repo, branch main, and the output artifact name SourceArtifact.
2. Build — choose AWS CodeBuild, select the project, and set SourceArtifact as the input. Output artifact will be BuildArtifact.
3. Skip the deploy stage for now — we’ll use the CLI approach later.

After creation, your pipeline will start automatically. Check the console — you should see the source and build stages turn green.

Pro tip: Use the visual editor to tweak the pipeline. You can add a manual approval gate between build and deploy for production control.

Step 5: Add deployment with the AWS CLI

To keep this lesson CLI-centered, deploy the artifact to an EC2 instance using CodeDeploy via CLI commands. First, create a deployment group in CodeDeploy (you’ll need an IAM role and an instance with the CodeDeploy agent). Then define a deployment with:

aws deploy create-deployment \
  --application-name my-app \
  --deployment-group-name my-deployment-group \
  --s3-location bucket=my-codepipeline-artifacts,key=BuildArtifact.zip,bundleType=zip

You’d then add a Deploy stage to your pipeline using the CLI. It’s a bit involved, but the key is that the pipeline automation is now complete: commit → build → test → deploy.

Pro tip: For serverless apps, use a Lambda deploy action instead of CodeDeploy. To see a full buildspec.yml for a Lambda app, check out our later lesson on Lambda deployments.

Compare options / when to choose what

AWS gives you several ways to implement CI/CD. Here’s a quick comparison to help you decide when to use CodePipeline versus other options.

When to choose CodePipeline:

• You’re already deep in the AWS ecosystem (using S3, EC2, Lambda).
• You need cross-account deployments or integration with IAM policies.
• You want a unified view of the whole release process in the AWS console.

When to think twice: - If your team lives in GitHub, GitHub Actions might be simpler. - If you need complex build matrices or custom plugins, Jenkins might be more flexible.

Variation 1: Use CodePipeline with AWS CodeBuild and ECS instead of EC2 — this gives you blue/green deployments with near-zero downtime.

Variation 2: Use CodePipeline with a Lambda function as a custom action to run a step that neither CodeBuild nor CodeDeploy handles natively (e.g., invoke a migration script).

Variation 3: Use CloudFormation with CodePipeline to deploy the infrastructure itself, not just the app. This is called infrastructure as code CI/CD.

Troubleshooting & edge cases

Even with the best setup, things go wrong. Here are the most common issues you’ll face and how to fix them.

1. Build fails because of missing dependencies

Symptom: The build logs show ModuleNotFoundError: No module named 'flask'. Fix: Ensure requirements.txt is committed to the repo and referenced in buildspec.yml. Also check the runtime version — you might have set Python 3.8 when your code needs 3.11.

2. Source stage doesn’t trigger on new commits

Symptom: You push to GitHub, but the pipeline doesn’t start. Fix: For GitHub, make sure you’ve configured a webhook in Settings. If you’re using polling (frequent interval), the pipeline might not start within a few seconds—wait up to 5 minutes. Also check the branch name matches the pipeline configuration.

3. Artifact files are missing in the deploy stage

Symptom: CodeDeploy says FileNotFoundError: dist/app.zip when you try to deploy. Fix: In your buildspec.yml’s artifacts section, specify the correct files. If you compile into a dist/ folder, add files: - 'dist/**/*'. Test by downloading the artifact manually from S3.

4. IAM permissions errors

Symptom: CodeBuild gets “Access Denied” when trying to write to S3 or pull from GitHub. Fix: Attach a proper IAM role to the CodeBuild project. The role must have permissions for S3 read/write and, if needed, a Secrets Manager secret for the GitHub token. For CodeDeploy, the EC2 instance must have the proper instance profile.

5. Deployment fails health checks

Symptom: The pipeline says Deploy failed because the instance didn’t pass health checks. Fix: Check the CodeDeploy agent logs on the instance (/var/log/aws/codedeploy-agent/codedeploy-agent.log). Common issues are expired TLS certs or wrong directory for the application files.

6. CodeBuild times out on pip install

Symptom: Build takes 10+ minutes and times out. Fix: Use a caching strategy—set the cache type to Amazon S3 in the CodeBuild project and cache ~/.cache/pip. Also, split your requirements into core and dev to speed up installs.

What you learned & what's next

You now understand the core of building a CI/CD pipeline with AWS CodePipeline: you have a mental model of stages and actions, you’ve seen how CodeBuild runs tests and packages your Python code, and you’ve walked through creating a pipeline using both the console and the CLI. You can apply this to your own projects by automating your build, test, and deploy steps for any Python application, and you know what to check when things go wrong.

Key takeaways: - CodePipeline orchestrates source, build, and deploy stages. - CodeBuild executes your buildspec and produces artifacts. - CodeDeploy handles deployments with health checks and rollback. - Every stage fail-stops the pipeline, giving you fast feedback.

The next lesson in this track covers monitoring and alerting your pipeline with CloudWatch Events and SNS notifications. That way, you’ll know automatically when a build fails, not by checking the console obsessively. You’ll also learn how to add manual approval actions for production releases—an essential step for enterprise teams.