Build a CI/CD Pipeline

You’ve written the code, tested it locally, and pushed it to GitHub. Then the deployment request comes in, and you find yourself running ssh, copying files, and restarting services by hand — again. Every manual step is a chance for a typo, a missed dependency, or a configuration drift that only shows up in production. This is exactly the pain that a CI/CD pipeline solves, and AWS CodePipeline is one of the fastest ways to build one without stitching together a dozen third-party tools.

In this lesson, you’ll learn to build a CI/CD pipeline with CodePipeline — from committing code to automatically building, testing, and deploying it — so you can ship features faster and with more confidence. By the end, you’ll have a working pipeline and a clear mental model for expanding it as your projects grow.

The problem this lesson solves

Manual deployments are fragile. When you deploy by hand, every release is a high-stakes ritual: you run tests locally (or forget to), you copy files to a server, and you pray the production environment matches your laptop. Even a single developer can spend hours on tasks that should take minutes, and the risk of human error grows with every step.

The deeper problem is lack of automation. Without an automated pipeline, you don’t get consistent feedback on whether your code actually works. A change that breaks a test might sit undetected until a customer hits it. And when you need to roll back, you’re left scrambling to remember what changed.

CI/CD — Continuous Integration and Continuous Delivery/Deployment — addresses this. The idea is simple: every code change triggers an automated workflow that builds, tests, and deploys your application. If any step fails, you know immediately. If it succeeds, your code is one push away from being live.

This lesson focuses on AWS CodePipeline, a managed service that orchestrates your entire release process. You don’t have to manage build servers or Jenkins master nodes; you define stages, and AWS handles the plumbing.

Core concept / mental model

Think of a CI/CD pipeline as an assembly line for your code. On a real assembly line, a car moves through stations: frame, engine, paint, and inspection. Each station performs a specific job, and if one station fails, the whole car is flagged. Your code follows the same path:

1. Source — where your code lives (e.g., GitHub, CodeCommit)
2. Build — compile, run tests, and package your application
3. Deploy — push the artifact to a target environment (e.g., EC2, ECS, Lambda)

In AWS, CodePipeline is the assembly track. It connects to source providers like GitHub or S3, triggers build providers like CodeBuild or Jenkins, and deploys to targets like Elastic Beanstalk or ECS. You define each stage once, and CodePipeline runs it every time your source changes.

This model gives you three core benefits:

• Automation — every step is triggered without human intervention
• Consistency — the same steps run on every commit, removing “works on my machine” surprises
• Visibility — you can see exactly where a release is, and if it fails, you know which stage to blame

Pro tip: You don’t need a full DevOps background to start. CodePipeline’s visual editor lets you click stages together, and the AWS CLI makes it scriptable. Start with a simple two-stage pipeline (source → build) and grow from there.

How it works step by step

Let’s break down the anatomy of a CodePipeline pipeline. You define stages, each containing actions that perform specific tasks. The flow is strict: a stage only starts after the previous stage completes successfully.

The three essential stages

1. Source stage — CodePipeline watches your repository for changes (e.g., every push to a branch). When it detects one, it pulls the latest code and stores it as an artifact — a zip of your repository that subsequent stages consume.

2. Build stage — Here you run your build and tests. In AWS, this is CodeBuild, which runs a build spec (buildspec.yml) in a managed, ephemeral container. It can run pytest, npm test, or any command you need. The output — say, a packaging of your app — becomes a new artifact.

3. Deploy stage — The final stage pushes your built artifact to a target environment. This could be a rolling update to EC2 via CodeDeploy, a new Lambda version, or a fresh ECS service. The choose-your-own-adventure part is here, because CodePipeline supports dozens of providers.

The whole process is event-driven: the source stage starts on a code push, and each stage is automatically triggered when the previous finishes. If a stage fails, the pipeline stops, and you get a notification (e.g., via CloudWatch Events or SNS).

How actions connect: artifacts and inputs/outputs

Each action can take input artifacts and produce output artifacts. For example, the source stage outputs a zip of your repo; the build action takes that zip as input, runs your build, and outputs a packaged artifact; the deploy action then consumes that artifact. You connect them by naming the outputs/inputs in the pipeline definition — or in the console, by just choosing the artifact from a dropdown.

Hands-on walkthrough

Let’s build a simple but complete pipeline with the AWS CLI. We’ll use GitHub as the source, CodeBuild to run pytest, and S3 as a pseudo-target (you’ll likely swap it for EC2 or ECS later).

Prerequisites

• An AWS account with AdministratorAccess (or at least permissions for IAM, CodePipeline, CodeBuild, and S3)
• The AWS CLI v2 installed and configured (aws configure)
• A GitHub repo with a Python app and a buildspec.yml file
• A GitHub personal access token with repo scope (to let CodePipeline access it)

Step 1: Create an S3 bucket for artifacts

CodePipeline stores artifacts in S3. Create a bucket and note its name:

aws s3api create-bucket \
  --bucket my-codepipeline-artifacts-$(date +%s) \
  --region us-east-1

Pro tip: Choose a region close to your user. The bucket name must be globally unique, so the timestamp suffix is a cheap trick.

Step 2: Add a buildspec.yml to your repo

In your GitHub repo, add a buildspec.yml file at the root. This tells CodeBuild what to do:

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.11
    commands:
      - pip install -r requirements.txt
  build:
    commands:
      - pytest  # run your tests

artifacts:
  files:
    - '**/*'
  discard-paths: no

Step 3: Define the pipeline with a JSON file

The pipeline definition is a JSON file that lists stages and actions. Create pipeline.json:

{
  "pipeline": {
    "name": "my-first-pipeline",
    "roleArn": "arn:aws:iam::ACCOUNT_ID:role/service-role/AWSCodePipelineServiceRole-us-east-1-my-first-pipeline",
    "artifactStore": {
      "type": "S3",
      "location": "my-codepipeline-artifacts-123"
    },
    "stages": [
      {
        "name": "Source",
        "actions": [
          {
            "name": "Source",
            "actionTypeId": {
              "category": "Source",
              "owner": "ThirdParty",
              "provider": "GitHub",
              "version": "1"
            },
            "configuration": {
              "Owner": "YOUR_GITHUB_USERNAME",
              "Repo": "YOUR_REPO_NAME",
              "Branch": "main",
              "OAuthToken": "YOUR_GITHUB_TOKEN"
            },
            "outputArtifacts": [
              {
                "name": "SourceArtifact"
              }
            ]
          }
        ]
      },
      {
        "name": "Build",
        "actions": [
          {
            "name": "Build",
            "actionTypeId": {
              "category": "Build",
              "owner": "AWS",
              "provider": "CodeBuild",
              "version": "1"
            },
            "configuration": {
              "ProjectName": "my-codebuild-project"
            },
            "inputArtifacts": [
              {
                "name": "SourceArtifact"
              }
            ]
          }
        ]
      }
    ]
  }
}

Step 4: Create the CodeBuild project

First, create a CodeBuild project that your pipeline will call. Use a simple IAM role for CodeBuild. You can do this via the console or CLI; here’s the CLI minimal:

aws codebuild create-project \
  --name my-codebuild-project \
  --source type=GITHUB,location=YOUR_REPO_URL \
  --environment type=LINUX_CONTAINER,computeType=BUILD_GENERAL1_SMALL,image=aws/codebuild/standard:6.0 \
  --service-role arn:aws:iam::ACCOUNT_ID:role/service-role/codebuild-service-role \
  --artifacts type=CODEPIPELINE

Pro tip: Setting artifacts.type=CODEPIPELINE tells CodeBuild to pass its output to the pipeline instead of uploading to S3 directly.

Step 5: Create the pipeline

Now create the pipeline from the JSON file:

aws codepipeline create-pipeline --cli-input-json file://pipeline.json

Step 6: Trigger the pipeline

Push a change to your GitHub repo (or use aws codepipeline start-pipeline-execution --name my-first-pipeline). Watch the pipeline run in the console.

Expected output (from the console or CLI):

Pipeline execution started.
Stage: Source - Succeeded
Stage: Build - Succeeded

The build stage logs should show your tests running and passing.

Compare options / when to choose what

CodePipeline isn’t the only CI/CD tool. Here’s how it stacks up against alternatives:

If your entire stack runs on AWS — which this track assumes — CodePipeline is the fastest path. It’s not ideal if you need to deploy to multiple clouds, because your pipeline logic would be tied to AWS. For that, a tool like GitLab CI or GitHub Actions with cloud-agnostic scripts gives more freedom.

Variations within CodePipeline:

• Source: GitHub, CodeCommit, S3, or ECR
• Build: CodeBuild, Jenkins (via plugins), or custom actions
• Deploy: EC2 (CodeDeploy), ECS, Lambda, or S3 (static sites)

Choose the simplest combination that meets your needs. A Lambda function, for example, might not need a deploy stage at all if you build and upload an artifact via a subsequent CFN or CLI call.

Troubleshooting & edge cases

Pipeline fails at the source stage: “Invalid GitHub token”

• Cause: The OAuth token is missing or expired.
• Fix: Regenerate the token in GitHub, update the pipeline, or connect via AWS’s built-in OAuth connector (which refreshes automatically).

Build stage fails with “Error: command not found: pytest”

• Cause: The buildspec instal phase didn’t install pytest.
• Fix: Ensure requirements.txt includes pytest and that your install commands are in the install phase. Check that you’re using a build image that matches your runtime (e.g., python:3.11).

Artifact not found in deploy stage

• Cause: The deploy action’s input artifact name doesn’t match the build output artifact.
• Fix: In the pipeline definition, set the inputArtifacts.name to exactly what your build configured (often BuildArtifact). You can see the names in the console — blank names are a common mistake.

Pipeline stuck in “In Progress”

• Cause: An action is waiting for a manual approval, or the stage is waiting for a retry.
• Fix: Check the “Approval” action’s URL/email. If you don’t need approval, remove that action. If it’s stuck due to permissions, verify the CodePipeline service role can access S3 and CodeBuild.

Pro tip: Use CloudWatch Events to notify you on pipeline failures. A simple SNS topic + Lambda function can give you immediate feedback via email or Slack.

What you learned & what's next

You’ve now built a CI/CD pipeline with CodePipeline. You understand the pain of manual deployments, the mental model of an assembly line, and the step-by-step mechanics of source, build, and deploy stages. You also know how to choose CodePipeline over other tools and can troubleshoot common hiccups.

You’ve met the learning objectives for this lesson:

• You can explain the core idea behind CodePipeline — it automates your release process by chaining source, build, and deploy stages.
• You completed a practical exercise — you created a pipeline that runs your tests on every push.

As a next step, consider connecting CodePipeline to a real deployment target: deploy a Lambda function via SAM, or roll out an EC2 fleet with CodeDeploy. That’s the natural continuation of this lesson in the AWS track.

Remember, the ultimate goal isn’t just automation — it’s ensuring that every change is delivered reliably, reproducibly, and quickly. With CodePipeline, you’ve taken a huge step toward that. Now go break things (in staging) and see your pipeline catch them!