Run Python Jobs with AWS Batch

You've spent hours perfecting a Python script on your laptop, only to realize the data you need to process has grown tenfold and your local machine is gasping for air. Waiting for heavy batch jobs to finish can stall your entire pipeline, and spinning up a dedicated server just for a few hours of compute feels like overkill. In this lesson, you'll learn how to run Python jobs with AWS Batch — a fully managed service that lets you execute containerized Python workloads at scale, without managing servers, and only pay for what you use.

The problem this lesson solves

Imagine you need to process millions of image files, run a complex data transformation, or train a machine learning model — these tasks aren't instant. Running them on your laptop ties up your machine, risks crashes, and can't easily scale. Even if you use a single EC2 instance, you're stuck with a fixed capacity, and you must handle setup, monitoring, and cleanup yourself.

AWS Batch solves this by providing a managed batch computing service that dynamically provisions the right amount of compute (EC2 instances, or even Fargate) based on the number and size of jobs in your queue. You package your Python code in a Docker image, define a job definition, and submit jobs to a queue. AWS Batch schedules them, runs them, and scales the underlying infrastructure automatically. This means you can run Python jobs with AWS Batch to handle anything from a single one-off script to a massive parallel workload, without babysitting infrastructure.

This lesson is a core step in your AWS Cloud & DevOps with Python journey. Once you master batch processing, you'll be able to build resilient, cost-effective data pipelines that scale effortlessly.

Core concept / mental model

Think of AWS Batch like a smart food delivery service for your compute tasks. You place an order (submit a job), the service decides which kitchen (compute environment) can prepare it fastest, and it scales up or down the number of chefs (EC2 instances or Fargate tasks) based on the volume of orders. You don't need to own the kitchen — you just describe what the dish looks like (your job definition) and let the service handle the logistics.

Here are the five core components:

• Job Definition — This is the recipe. It specifies the Docker image, command to run, resource requirements (vCPUs, memory), and environment variables.
• Job Queue — This is the waiting line. Jobs are submitted to a queue, and the scheduler picks them up based on priority.
• Compute Environment — This is the kitchen. It defines where jobs run (EC2 or Fargate), and how many resources are available. It can be managed (AWS Batch creates and scales instances for you) or unmanaged (you provide your own instances).
• Job — This is a single order. It's a unit of work (e.g., running your Python script) that's submitted to a queue.
• Scheduler — This is the brain. It evaluates the queue, chooses which jobs to run, and allocates compute resources accordingly.

Pro tip: AWS Batch is often compared to a simpler version of Kubernetes for batch workloads. If you don't need Kubernetes' complex orchestration features, AWS Batch gives you a serverless-like experience with less overhead.

How it works step by step

To run a Python job with AWS Batch, you follow a series of steps that transform your code into a managed, scalable execution:

1. Containerize your Python script — First, you need to package your code into a Docker image. This makes it portable and ensures consistent dependencies across any environment.
2. Push the image to Amazon ECR (Elastic Container Registry) — AWS Batch needs a place to pull your image from. ECR is the natural choice, though you could use Docker Hub.
3. Create a compute environment — This tells AWS Batch what type of infrastructure to use. You can pick EC2 for predictable, long-running jobs or Fargate for a serverless model.
4. Create a job queue — This is the bucket where your jobs wait to be processed. You associate the queue with a compute environment.
5. Define a job definition — This is the blueprint for your job. It references your Docker image, the command to run, and resource limits.
6. Submit a job — Use the AWS CLI, SDK, or console to send the job to the queue.
7. Monitor and scale — AWS Batch runs the job, monitors its status, and scales compute resources up or down based on the queue depth.

Each step builds on the previous one. Once you've done this once, the process becomes second nature — and you can even automate it with CloudFormation or Terraform, as you'll learn later in this track.

Hands-on walkthrough

Let's get your hands dirty. We'll create a simple Python job that prints a message, then we'll scale it up to process a list of files. This hands-on exercise will solidify the core concepts.

Prerequisites

• AWS CLI installed and configured.
• Docker installed.
• Basic knowledge of Python and the AWS console.
• Make sure you have the boto3 library for the SDK examples.

Step 1: Write and containerize your Python script

Create a file called hello_batch.py:

#!/usr/bin/env python3
import sys

def main():
    if len(sys.argv) > 1:
        name = sys.argv[1]
    else:
        name = "World"
    print(f"Hello from AWS Batch, {name}!")
    # Simulate work
    for i in range(10):
        print(f"Working... {i}")
    print("Done.")

if __name__ == "__main__":
    main()

Now create a Dockerfile:

FROM python:3.11-slim
COPY hello_batch.py /app/hello_batch.py
WORKDIR /app
ENTRYPOINT ["python", "hello_batch.py"]

Build and push the image to ECR:

# Authenticate Docker to ECR (replace region and account id)
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com

# Create a repository
echo "aws ecr create-repository --repository-name hello-batch --region us-east-1"

# Build
 docker build -t hello-batch .

# Tag
docker tag hello-batch:latest 123456789012.dkr.ecr.us-east-1.amazonaws.com/hello-batch:latest

# Push
docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/hello-batch:latest

Step 2: Create compute environment and job queue (using AWS CLI)

Here's how to create a managed EC2 compute environment and a queue:

# Create compute environment
aws batch create-compute-environment \
    --compute-environment-name my-ec2-env \
    --type MANAGED \
    --state ENABLED \
    --compute-resources type=EC2,minvCpus=0,maxvCpus=16,desiredvCpus=0,instanceTypes=optimal,subnets=subnet-abc123,securityGroupIds=sg-abc123

# Create job queue
aws batch create-job-queue \
    --job-queue-name my-job-queue \
    --state ENABLED \
    --priority 1 \
    --compute-environment-order order=1,computeEnvironment=my-ec2-env

The compute environment may take a few minutes to become VALID before you can attach it to a queue.

Step 3: Define the job definition

aws batch register-job-definition \
    --job-definition-name hello-batch-job \
    --type container \
    --container-properties '{
      "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/hello-batch:latest",
      "vcpus": 1,
      "memory": 512,
      "command": ["Ref::name"]
    }' \
    --parameters '{"name": "AWS"}'

Step 4: Submit a job

aws batch submit-job \
    --job-name my-first-job \
    --job-queue my-job-queue \
    --job-definition hello-batch-job \
    --parameters name="PythonDeveloper"

Step 5: Monitor status

aws batch describe-jobs --jobs <job-id>

You'll see statuses like SUBMITTED, PENDING, RUNNABLE, STARTING, RUNNING, and SUCCEEDED. Eventually, check the CloudWatch logs for the output.

Bringing it together with Python and boto3

You can also submit jobs from your own Python code using boto3. Here's an example:

import boto3

# Create a Batch client
client = boto3.client('batch', region_name='us-east-1')

response = client.submit_job(
    jobName='my-python-job',
    jobQueue='my-job-queue',
    jobDefinition='hello-batch-job',
    parameters={'name': 'from-python'}
)

job_id = response['jobId']
print(f'Job submitted: {job_id}')

# Wait for it to complete (polling)
while True:
    status = client.describe_jobs(jobs=[job_id])['jobs'][0]['status']
    if status in ['SUCCEEDED', 'FAILED']:
        print(f'Job finished with status: {status}')
        break
    import time
    time.sleep(5)

Expected output: The job will print something like:

Hello from AWS Batch, from-python!
Working... 0
Working... 1
...
Done.

Compare options / when to choose what

AWS Batch isn't the only way to run batch workloads on AWS. Let's compare it with some alternatives:

Variations: - Use Fargate as the compute environment for a serverless batch experience — you don't manage EC2 instances at all. - Use array jobs to run thousands of similar tasks (like processing multiple files) in parallel with a single submit. - Combine with AWS Step Functions to orchestrate batch jobs with other services in a pipeline.

For most batch processing tasks in a DevOps pipeline, AWS Batch is the sweet spot: it's a managed, cost-effective way to run Python jobs at scale without reinventing the wheel.

Troubleshooting & edge cases

Even with AWS Batch, things can go wrong. Here are common pitfalls and how to fix them:

• Job stays in RUNNABLE but never starts. This often means the compute environment is at its max vCPUs, or there's a subnet/security group misconfiguration. Check the compute environment status and logs.
• FAILED with CannotPullContainerError. The image name or ECR repository might be wrong, or the IAM role doesn't have permissions to pull from ECR. Double-check your image URI and the executionRoleArn in the job definition.
• The job runs but crashes immediately. Check CloudWatch logs for Python errors. Make sure your entrypoint and command are correct — especially if you're passing parameters.
• The compute environment never becomes VALID. This can happen if the IAM role for AWS Batch doesn't have the required permissions, or the subnets are wrong. Verify the service-linked role AWSServiceRoleForBatch exists.
• You're getting charged even when idle. If you use a managed EC2 environment, ensure minvCpus=0 so it scales down to zero when no jobs are running.
• Fargate jobs require a platform version and execution role. For Fargate compute environments, you need to specify platformVersion=latest and provide an executionRoleArn with ECR pull permissions.

Pro tip: Always set maxvCpus to a number that matches your budget. AWS Batch will happily scale up to the max and cost you money if you forget.

What you learned & what's next

You've now mastered the core of running Python jobs with AWS Batch. You understand the problem of managing batch workloads manually, and you've built a mental model of AWS Batch components. You know the step-by-step process from containerizing your script to submitting and monitoring jobs. You've seen how to compare AWS Batch with alternatives, and you've learned the most common troubleshooting pitfalls.

Specifically, you can:

• Explain why AWS Batch is the go-to service for containerized Python batch processing.
• List the core components: job definition, job queue, compute environment, and jobs.
• Create a compute environment and job queue, register a job definition, and submit a job using the AWS CLI and boto3.
• Choose between AWS Batch, Lambda, and other alternatives based on the use case.
• Debug common issues like stuck jobs, container pull errors, and scaling costs.

This is a foundational skill for anyone building data pipelines or DevOps automation. Next in this track, you'll explore how to integrate AWS Batch with other AWS services, such as triggering jobs from S3 events or orchestrating with Step Functions. That will bring you closer to building fully automated, event-driven data processing systems.