Monitor S3 Access Logs

You deployed a critical S3 bucket, set up lifecycle rules, and locked down the bucket policy — but you have no idea who is actually accessing your objects, from where, or how often. Without visibility into S3 access patterns, a leaked credential or a misconfigured policy can go unnoticed for weeks, turning a minor mistake into a major breach. This lesson shows you how to monitor S3 with access logs so you can detect anomalies, audit compliance, and respond to security incidents before they escalate.

The problem this lesson solves

S3 is a cornerstone of modern cloud storage, but its power is also its danger. By default, S3 does not log every request to your bucket. That means you are flying blind when it comes to answering questions like: Who listed the bucket? Did that IP range really download our backup? Is someone brute-forcing object keys?

Without access logs, you cannot prove who accessed what, when, or from where — making compliance audits (think SOC 2, GDPR, HIPAA) nearly impossible. Worse, you miss early warning signs of a compromised credential or an over-permissive policy. An attacker might sneak in, exfiltrate data, and erase trace logs — all without you ever knowing.

The solution is server access logging: a native feature that records every request made to your bucket, delivering detailed log files to a target bucket of your choice. But enabling it is only half the battle. You also need a reliable way to monitor S3 with access logs — parse, query, alert, and act on the data.

Core concept / mental model

Think of your S3 bucket as a fortress. The bucket policy and IAM roles are your guards at the gate. But even the best guards cannot write a report of everything they see — unless you give them a pen and paper. S3 access logs are that pen and paper.

Every time a request is made (GET, PUT, DELETE, LIST, or even a denied attempt), S3 records a log entry that includes:

• Requester (the AWS account ID or IAM role ARN)
• IP address (source IP of the request)
• Operation (e.g., REST.GET.OBJECT)
• Object key (the file accessed)
• Response status (200, 403, 404, etc.)
• Timestamp (when the request occurred)

These logs are stored as object files in a separate S3 bucket (the "target" bucket). They are typically delivered within a few hours and are batched, not real-time.

Key mental model: Access logs are after-the-fact records, not real-time alerts. For immediate detection, combine them with AWS CloudTrail (for API-level events) and Amazon GuardDuty (for anomaly detection). Access logs give you granular object-level detail that CloudTrail often misses.

How it works step by step

Enabling and using S3 access logs follows a predictable flow. Here’s the cause-and-effect sequence:

1. Create a target bucket (or use an existing one) to store the logs. It is best practice to keep logs in a separate bucket with restricted access — only the log delivery service should write to it.
2. Enable server access logging on the source bucket: specify the target bucket and a prefix (e.g., logs/).
3. Verify delivery after a few hours. The first logs may take up to 2–3 hours to appear.
4. Set up parsing and analysis — either using AWS Athena (SQL queries directly on S3) or by writing a small Python script to parse the log files.
5. Define alerts based on suspicious patterns (e.g., repeated 403s, unknown IPs, large data transfer).

Each step is simple, but skipping one (like forgetting to restrict the target bucket) can undermine your entire monitoring strategy.

Hands-on walkthrough

Let’s put this into practice. We’ll use the AWS CLI and Python to enable logging, simulate access, and parse the logs.

1. Enable server access logging via AWS CLI

First, create a target bucket (if you don’t have one) and enable logging on your source bucket. Store the bucket names in environment variables to avoid repetition.

# Create a target bucket (must be in the same region as source bucket)
aws s3api create-bucket --bucket my-logs-bucket --region us-east-1

# Enable access logging on the source bucket
aws s3api put-bucket-logging \
  --bucket my-source-bucket \
  --bucket-logging-status '{"LoggingEnabled": {"TargetBucket": "my-logs-bucket", "TargetPrefix": "logs/"}}'

# Verify logging is enabled
aws s3api get-bucket-logging --bucket my-source-bucket

Note: Replace my-source-bucket with your actual bucket name. The target bucket must be in the same AWS region as the source bucket, or log delivery will fail.

2. Simulate traffic to generate logs

Access logs only appear when there is activity. Let’s generate some.

# Upload a file (this triggers a PUT request)
echo "Hello S3 logs" | aws s3 cp - s3://my-source-bucket/test.txt

# Download the file (this triggers a GET request)
aws s3 cp s3://my-source-bucket/test.txt ./downloaded.txt

# Attempt an unauthorized access (this triggers a 403)
aws s3 ls s3://my-source-bucket --no-sign-request

3. Write a Python script to parse access logs

Access logs are written in a space-delimited format but with quoted strings for fields that contain spaces. Use Python’s csv module with a custom delimiter to parse them.

import csv
import os
from datetime import datetime

# Path to a sample log file (adjust to your downloaded log)
log_file = "logs/2025-01-01-00-00-00-ABCDEFGH"

# S3 access log format fields (in order)
FIELDS = [
    'bucket_owner', 'bucket', 'time', 'remote_ip', 'requester',
    'request_id', 'operation', 'key', 'request_uri', 'http_status',
    'error_code', 'bytes_sent', 'object_size', 'total_time', 'turnaround_time',
    'referrer', 'user_agent', 'version_id', 'host_id', 'tls_version',
    'cipher_suite'
]

with open(log_file, 'r') as f:
    reader = csv.DictReader(f, fieldnames=FIELDS, delimiter=' ')
    for row in reader:
        # Print only successful object retrievals
        if row.get('operation') == 'REST.GET.OBJECT' and row.get('http_status') == '200':
            print(f"[{row['time']}] GET {row['key']} from {row['remote_ip']}")

Expected output (after you download an actual log file from your target bucket):

[01/Jan/2025:00:00:00 +0000] GET test.txt from 203.0.113.10

4. Query logs with Amazon Athena (optional but powerful)

For production-scale, use Athena to run SQL on your S3 logs. First, create a table in Athena using the AWS Glue Data Catalog, or use the following quick table definition:

CREATE EXTERNAL TABLE IF NOT EXISTS s3_access_logs (
  bucket_owner STRING,
  bucket STRING,
  request_time STRING,
  remote_ip STRING,
  requester STRING,
  request_id STRING,
  operation STRING,
  key STRING,
  request_uri STRING,
  http_status INT,
  error_code STRING,
  bytes_sent BIGINT,
  object_size BIGINT,
  total_time INT,
  turnaround_time INT,
  referrer STRING,
  user_agent STRING,
  version_id STRING,
  host_id STRING,
  tls_version STRING,
  cipher_suite STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ' '
STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://my-logs-bucket/logs/';

Then query for suspicious 403s:

SELECT remote_ip, COUNT(*) AS denied_count
FROM s3_access_logs
WHERE http_status = 403
GROUP BY remote_ip
ORDER BY denied_count DESC
LIMIT 10;

Compare options / when to choose what

S3 access logs are not your only monitoring tool. Here’s how they stack up against alternatives.

When to choose what:

• Use S3 access logs when you need complete object-level history for forensics or compliance. They are the only option that records every GET and PUT.
• Use CloudTrail if you only care about who changed permissions or deleted the bucket? CloudTrail gives you management-plane visibility but misses data-plane actions like object GETs.
• Use event notifications for real-time reactions (e.g., virus scanning on upload), but they only trigger on specific events, not all requests.
• Use GuardDuty as a top-level alarm — it uses machine learning to spot unusual patterns, but it won’t give you raw access logs.

Pro tip: For a complete security posture, combine S3 access logs with CloudTrail and GuardDuty. Each covers a blind spot the others miss.

Troubleshooting & edge cases

Common issues when monitoring S3 with access logs — and how to fix them.

Logs are not appearing after several hours

• Cause: The target bucket is in a different region than the source bucket.

• Fix: Create a new target bucket in the same region, or move the existing one.

• Cause: The target bucket policy is missing. S3 needs write permission to the target bucket.

• Fix: Add a bucket policy to the target that allows s3:PutObject from the log delivery service. AWS provides a standard policy template in the console.

Logs contain only - for certain fields

• Cause: Some fields (like requester for anonymous requests) are intentionally blank.
• Fix: No action needed; it’s normal.

You see 403 errors but no logs for denied requests

• Cause: By default, S3 access logging logs all requests, including denied ones. If you don't see them, check that the bucket policy on the source bucket does not block the logging service itself.
• Fix: Review the source bucket’s bucket policy to ensure it allows s3:PutObject to the target bucket. Also, if you have a bucket policy that denies all access (except for a few principals), the log delivery service might be rejected. Add an explicit allow for the logging service’s canonical user ID.

Logging causes a lot of extra storage costs

• Cause: Log files can be large and numerous.
• Fix: Set up lifecycle rules on the target bucket to transition logs to S3 Glacier after 30 days and permanently expire after 90 days.

You suspect real-time attacks but logs are delayed

• Cause: Access logs are not real-time.
• Fix: Use CloudWatch Events or event notifications for immediate alerts, but rely on access logs for post-incident analysis.

What you learned & what's next

You now understand how to monitor S3 with access logs: you can enable server access logging, parse, query, and alert on the collected data. You learned that access logs give you object-level visibility that CloudTrail doesn’t, but they arrive with a delay — so combine them with other tools for full coverage.

You also connected this skill to your broader cloud security toolkit: access logs are essential for compliance and forensic investigations, and you now know how to set up the plumbing.

Next in this track: Dive into AWS CloudTrail to monitor API-level activity. You’ll learn how to detect when someone alters your bucket policy or IAM roles — closing the gap between data-plane logs (what you just did) and management-plane trails. Together, they give you a complete audit trail for your AWS environment.