Work with boto3 for AWS automation

Manually clicking through the AWS Console to create S3 buckets, snapshot EC2 instances, or rotate IAM keys is a productivity killer — and worse, it's a reliability risk when someone forgets a step or applies the wrong configuration. If you're a DevOps engineer automating cloud infrastructure, you need a programmatic, repeatable way to manage AWS resources. That's exactly what boto3, the official AWS SDK for Python, gives you. This lesson shows you how to work with boto3 for AWS automation — from the core mental model to practical, runnable scripts that you can adapt immediately.

The problem this lesson solves

Imagine you need to provision a new S3 bucket for every feature branch, snapshot a database before a risky migration, or clean up old EC2 instances. Doing this by hand doesn't scale and invites human error. The problem: cloud operations are tedious, error-prone, and impossible to audit when done manually. The solution: using Python with boto3 to define infrastructure and operations as code — so the same script runs the same way every time, and you can version it, review it, and reuse it.

Pro tip: boto3 is not just a wrapper around the AWS API. It handles retries, pagination, and session management for you — things that would take hours to implement yourself.

Core concept / mental model

Think of boto3 as your Python conversation layer with AWS. Instead of clicking buttons, you send request objects and receive response objects. The SDK translates your Python calls into HTTP requests to AWS's service endpoints, and then parses the JSON responses back into Python dictionaries and lists.

Here's the mental model: 1. Client vs. Resource — boto3.client() gives you a low-level, service-specific client (e.g., s3, ec2) with get_, create_, delete_ methods. boto3.resource() gives a higher-level, object-oriented interface where you work with Bucket, Instance, etc. 2. Session & Credentials — boto3 uses a credential chain: environment variables, shared credential file, IAM roles, or AWS config. The default chain works out-of-the-box for most setups. 3. Pagination — AWS APIs often return limited results per call. boto3 can automatically loop through pages using the PaginationConfig parameter or the paginator objects. 4. Regions — Every client is bound to a region (e.g., us-east-1). Some resources are global (like IAM), but most are regional.

import boto3

# Low-level client
s3_client = boto3.client('s3', region_name='us-east-1')
print(type(s3_client))  # <class 'botocore.client.S3'>

# High-level resource
s3_resource = boto3.resource('s3', region_name='us-east-1')
print(type(s3_resource))  # <class 'boto3.resources.factory.s3.ServiceResource'>

Output:

<class 'botocore.client.S3'>
<class 'boto3.resources.factory.s3.ServiceResource'>

How it works step by step

Working with boto3 follows a predictable sequence — once you learn it, you can apply the same pattern to any AWS service.

1. Install and import boto3 — pip install boto3. Then import boto3 in your script.
2. Configure credentials — Either set environment variables like AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or use the shared credentials file at ~/.aws/credentials. For security, prefer IAM roles when running in EC2 or Lambda.
3. Create a client or resource — Choose based on whether you need granular control (client) or convenience (resource).
4. Call the method — For example, s3.create_bucket(Bucket='name', CreateBucketConfiguration={'LocationConstraint': 'us-west-2'}).
5. Handle the response — The response is a dict with a 'ResponseMetadata' key, plus service-specific data. Always check the HTTPStatusCode.
6. Iterate with paginators when needed — For lists (e.g., all S3 buckets or EC2 instances), use paginator.paginate() to loop through all results.

Hands-on walkthrough

Let's build a real automation script: list and clean up old EC2 snapshots. This is a common DevOps task to save cost.

Example 1: List EC2 instances and their states

import boto3

def list_instances(region='us-east-1'):
    ec2 = boto3.client('ec2', region_name=region)
    response = ec2.describe_instances()
    instances = []
    for reservation in response['Reservations']:
        for instance in reservation['Instances']:
            instances.append({
                'Id': instance['InstanceId'],
                'State': instance['State']['Name'],
                'Type': instance['InstanceType']
            })
    return instances

print(list_instances())

Expected output (example):

[{'Id': 'i-0abc1234567890def', 'State': 'running', 'Type': 't3.micro'},
 {'Id': 'i-0fedcba9876543210', 'State': 'stopped', 'Type': 'm5.large'}]

Example 2: Snapshot all EBS volumes attached to running instances

import boto3

def snapshot_running_volumes(region='us-east-1'):
    ec2 = boto3.client('ec2', region_name=region)
    instances = ec2.describe_instances(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
    snapshots = []
    for reservation in instances['Reservations']:
        for instance in reservation['Instances']:
            for block in instance['BlockDeviceMappings']:
                vol_id = block['Ebs']['VolumeId']
                resp = ec2.create_snapshot(
                    VolumeId=vol_id,
                    Description=f"Snapshot for {instance['InstanceId']} on {datetime.utcnow()}"
                )
                snapshots.append(resp['SnapshotId'])
    return snapshots

from datetime import datetime
print(snapshot_running_volumes())

Note: This script requires ec2:CreateSnapshot and ec2:DescribeInstances permissions.

Example 3: Use a paginator to list all S3 buckets

import boto3

s3 = boto3.client('s3')
paginator = s3.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket='my-bucket', PaginationConfig={'PageSize': 5}):
    for obj in page.get('Contents', []):
        print(obj['Key'], obj['Size'])

Compare options / when to choose what

When to choose what: Use boto3 client when you need detailed control or when the resource interface doesn't exist. Use resource for frequent, simple operations (like S3). For infrastructure provisioning that's long-lived, consider Terraform — but for day-to-day operations, boto3 is your Swiss Army knife.

Troubleshooting & edge cases

1. No credentials found

botocore.exceptions.NoCredentialsError: Unable to locate credentials

Fix: Set environment variables or configure ~/.aws/credentials correctly. Verify with aws sts get-caller-identity.

2. Permission denied

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListBuckets operation

Fix: Ensure your IAM policy includes the required actions (e.g., s3:ListAllMyBuckets).

3. Wrong region for S3 bucket creation

Creating an S3 bucket in us-east-1 doesn't require LocationConstraint, but other regions do. If you specify one for us-east-1, you'll get IllegalLocationConstraintException. Use conditional logic.

import boto3
s3 = boto3.client('s3', region_name='us-east-1')
try:
    s3.create_bucket(Bucket='my-bucket')
except ClientError as e:
    # Handle 409 BucketAlreadyOwnedByYou
    pass

4. Pagination missed items

If you only fetch one page, you might miss resources. Always use paginators or loop until IsTruncated is false.

5. Rate limiting

AWS throttles API calls. boto3 retries automatically, but you should still add exponential backoff for robustness.

What you learned & what's next

You now understand how to work with boto3 for AWS automation — the core client/resource model, how to authenticate, and how to automate common tasks like listing instances, snapshotting volumes, and paginating results. You've completed a practical exercise that can be extended to manage your own infrastructure.

Next step: Now that you can automate AWS with Python, the next lesson in this track will show you how to build reusable boto3 helper modules — wrapping common operations (like instance start/stop or S3 upload/download) into clean, testable functions you can use across your DevOps tooling. You'll also learn how to integrate these helpers into CI/CD pipelines. Until then, try extending today's script to automatically delete snapshots older than 7 days — a classic cost-saving automation.