Set up CloudTrail audit trails

You’ve hardened your IAM policies, encrypted your data with KMS, and locked down your container images. But when something goes wrong in your AWS account — an API call that shouldn’t have happened, a role that was assumed from an unexpected IP, or a resource deleted at 3 AM — do you have any way to prove what happened, when, and by whom? Without an audit trail, you’re flying blind. That’s exactly the pain this lesson solves: setting up CloudTrail audit trails so you can answer the most important security question — “what changed?” — with confidence and evidence.

The problem this lesson solves

Cloud environments are dynamic by design, and that dynamism is also their greatest security risk. Users, roles, and services are constantly making API calls: launching EC2 instances, modifying S3 bucket policies, assuming cross-account roles, deleting CloudWatch log groups. Each of those actions can be legitimate — or the first step in an attack. Without a centralized, tamper-evident audit trail, you cannot:

• Detect unauthorized or anomalous activity in a timely manner.
• Investigate security incidents after the fact.
• Comply with regulatory frameworks like SOC 2, PCI DSS, HIPAA, or ISO 27001 that mandate logging and monitoring.

Imagine a developer accidentally leaves an S3 bucket public. A week later, you discover that sensitive customer data was exfiltrated. Without CloudTrail, you have no way to know exactly which IAM user made the bucket public, when it happened, or which IP address was used. With CloudTrail, you can reconstruct the entire event.

The reality is: auditing is not optional in production. CloudTrail is AWS’s built-in answer — but it does not come fully configured out of the box. This lesson gives you a practical, repeatable process to set up CloudTrail audit trails that actually help you sleep at night.

Core concept / mental model

Think of CloudTrail as the flight recorder for your AWS account. Every API call — whether made via the AWS Management Console, the CLI, an SDK, or another AWS service — generates an event. CloudTrail captures those events and writes them to a durable, queryable destination.

Here’s the mental model in three layers:

1. Event source: Any principal (IAM user, role, or service) makes an API call. The call hits AWS’s public endpoints.
2. CloudTrail engine: The service records the event, including the identity, time, source IP, request parameters, and response elements.
3. Delivery destination: CloudTrail writes the logs either to an S3 bucket (the classic choice) and optionally to CloudWatch Logs for real-time monitoring and alarming.

Pro tip: CloudTrail is not a security control in isolation — it’s the evidence layer that makes your other controls (IAM, VPC security groups, WAF) measurable. No trail, no proof.

Two important facts to keep in mind:

• Management events — the bread and butter — include actions like ec2:RunInstances, s3:PutBucketPolicy, iam:CreateUser, and sts:AssumeRole. They are logged by default when you enable a trail.
• Data events — like S3 object-level operations (s3:GetObject) or Lambda function invocations — are not logged by default. You must explicitly add them, and they come at an extra cost.

This is a critical distinction: a default CloudTrail trail gives you a management audit trail, but if you need visibility into data plane access, you must configure data events separately.

How it works step by step

Let’s walk through the process of setting up a CloudTrail audit trail that is both secure and useful. You can do this via the AWS Management Console, the AWS CLI, or Infrastructure as Code (like Terraform or CloudFormation). The steps below apply to all methods.

Step 1: Decide on a multi-region trail

A single-region trail only logs events from that one region. For a comprehensive audit trail, create a multi-region trail. This ensures you capture activity across all current and future AWS regions — essential for global teams and incident response.

Step 2: Create an S3 bucket for log storage

CloudTrail needs a destination bucket. Avoid using an existing application bucket — create a dedicated one. The bucket must have block public access turned on, and CloudTrail will automatically add a bucket policy that allows only CloudTrail to write logs. You can enable SSE-S3 (default) or SSE-KMS for stronger encryption.

Step 3: Enable the trail (with default management events)

When you create a trail, you can:

• Keep management events enabled for all read and write actions.
• Optionally enable data events for S3 and Lambda.
• Optionally enable insights events for anomaly detection (much more expensive).

Step 4: Deliver logs to CloudWatch Logs (optional but recommended)

Instead of only storing logs in S3, you can also have CloudTrail send events to CloudWatch Logs. This enables you to create metric filters and alarms — for example, alert on iam:CreateUser or sts:AssumeRole with a suspicious source IP. This is the difference between a passive archive and an active early-warning system.

Step 5: Validate log file integrity

CloudTrail can sign log files so you can detect if they were tampered with after delivery. Enable log file validation — it adds a digest file that you can verify later, an absolute must for compliance and forensic credibility.

Hands-on walkthrough

Let’s get practical. You’ll create a CloudTrail trail that logs events across all regions and delivers them to S3 and CloudWatch Logs.

Prerequisites

• An AWS account and credentials with permissions to create CloudTrail trails, S3 buckets, and IAM roles.
• AWS CLI installed and configured (aws configure).

Option A: Create a trail via the AWS CLI

The fastest way is to use the AWS CLI. First, create the S3 bucket. Remember: S3 bucket names are globally unique.

aws s3api create-bucket \
  --bucket my-cloudtrail-logs-2025 \
  --region us-east-1

Next, create the trail. Use --is-multi-region-trail to capture all regions, and attach the bucket as the destination.

aws cloudtrail create-trail \
  --name my-audit-trail \
  --s3-bucket-name my-cloudtrail-logs-2025 \
  --is-multi-region-trail \
  --enable-log-file-validation

Finally, start logging.

aws cloudtrail start-logging \
  --name my-audit-trail

That’s it! CloudTrail will now begin recording management events. To see the logs, list the S3 bucket:

aws s3 ls my-cloudtrail-logs-2025 --recursive

You should see a folder structure like AWSLogs/123456789012/CloudTrail/us-east-1/2025/01/15/.

Option B: Adding CloudWatch Logs delivery

To route events to CloudWatch Logs for real-time monitoring, you need to create a log group and an IAM role that CloudTrail assumes.

First, create the CloudWatch Logs log group:

aws logs create-log-group --log-group-name /aws/cloudtrail/my-audit-trail

Then, in the AWS Console, edit your trail and under “CloudWatch Logs” select the log group and create a new IAM role (or use an existing one with the correct permissions). CloudTrail will generate a role policy for you.

Alternatively, you can use the console wizard for the whole setup — it’s more guided but involves more clicking.

Validate the trail

After a few minutes, trigger a test event (e.g., create an S3 bucket) and then query CloudTrail using the CLI:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateBucket

You’ll see an event with details like the IAM user, source IP, and timestamp.

Pro tip: Use lookup-events for quick debugging, but for deep analysis (like “who assumed this role in the last 90 days”), use Amazon Athena with the CloudTrail table — it’s the standard pattern.

Compare options / when to choose what

CloudTrail offers different ways to configure your audit trails. Here’s a quick comparison:

For the majority of security-conscious teams, a multi-region trail with management events and log file validation is the right starting point. Add data events only for S3 buckets that contain sensitive data — otherwise, the cost can add up quickly (you are billed per event). Insights events are powerful but should be treated as an advanced feature — they double the number of events you’re billed for.

Troubleshooting & edge cases

Even with a straightforward service like CloudTrail, a few common issues can trip you up:

1. Logs are not appearing in S3

If you create a trail but no logs arrive after 15 minutes, the most common cause is a hash mismatch in the bucket policy — either you didn’t use the exact bucket policy CloudTrail generated, or you manually edited it and broke a condition. Fix: delete the trail and recreate it in the console, letting CloudTrail manage the policy.

2. CloudTrail cannot deliver to S3 due to bucket permissions

You must ensure the S3 bucket has a policy that allows cloudtrail.amazonaws.com to s3:PutObject and s3:GetBucketAcl. If you see “Trail was created but logging failed,” check the bucket policy.

3. You are being charged a lot for data events

Data events can balloon your bill if you enable them on all S3 buckets. Mitigation: Enable data events only on specific buckets (by ARN) or use event selectors to filter to write operations only.

4. Logs are missing management events from the console

Remember: the console itself makes API calls on your behalf. For example, visiting the S3 console generates a ListBuckets call. If you don’t see those events, your trail is not capturing read events — go to the trail’s event selector settings and ensure “Read” is enabled.

5. tampered or missing digest files

If you enabled log file validation but digests are missing, check the bucket policy allows GetObject on the digest path (usually AWSLogs/<account>/CloudTrail-Digest).

What you learned & what's next

You now know how to set up CloudTrail audit trails that record every API call across your AWS account — from enabling a multi-region trail to delivering logs to CloudWatch and validating file integrity. You understand the crucial difference between management and data events, and you have a practical method for creating a trail via the CLI or console.

More importantly, you’ve connected this to your cloud security posture: without these trails, you cannot detect or investigate incidents. You’ve applied Set up CloudTrail audit trails in a hands-on exercise — and now you’re ready to move to the next lesson in this track, where you’ll use those logs to build alerting and detective controls.

Next step: In the upcoming lesson, you’ll harness CloudTrail logs with CloudWatch alarms to automate first-response actions when high-risk API calls happen. Keep your trail running — you’ll need it.