Collect Python Logs with CloudWatch

You’ve built a Python app, deployed it to AWS, and it works — until it doesn’t. When a request fails at 2 AM, your first instinct is to check the logs. But where are they? If you’re SSHing into EC2 instances and tailing files, you’re losing time and missing critical context. This lesson shows you how to collect Python logs with CloudWatch Logs so every log line from your app lands in one searchable, centralized place — ready for debugging, alerting, and auditing. By the end, you’ll have a repeatable pattern to ship logs from any Python service to AWS, without third-party agents or complex setup.

The problem this lesson solves

Traditional logging in Python often means writing to a local file or printing to stdout. That approach breaks in the cloud for three reasons:

• Instances are ephemeral. EC2 instances get replaced; logs stored locally vanish with them.
• Distributed apps produce scattered logs. With multiple services running, you need to correlate logs across hosts.
• Debugging is reactive and slow. Tailing log files over SSH doesn’t scale and doesn’t give you metrics or alerts.

Imagine you have a Django app on EC2 and an API on Lambda. Without a central log store, you’d have to SSH into each instance and hunt for the relevant file. In a containerized world, that’s even worse — containers are destroyed and recreated constantly. The pain is real: you need log collection that is automatic, centralized, and searchable.

CloudWatch Logs solves this by acting as a central repository for logs from all your AWS resources and custom applications. But the challenge is getting Python logs into it — and that’s exactly what this lesson teaches.

Core concept / mental model

Think of CloudWatch Logs as a post office for your log messages. Your Python app is a writer that drops letters (log lines) into a mailbox. The mailbox is a log group — a container for related logs, like a folder. Within each log group, you have log streams — like individual mailboxes for each source (e.g., a specific EC2 instance or container).

Here’s the mental model in a nutshell:

• Log group — A named collection of log streams, typically representing an application or a component (e.g., /myapp/backend).
• Log stream — A sequence of log events from a single source (e.g., a single EC2 instance or a single Lambda invocation).
• Log event — A single log entry with a timestamp and message.

The key insight: you don’t send logs directly to CloudWatch from your Python code (unless you’re in Lambda). Instead, you use the CloudWatch Logs agent (or the unified CloudWatch agent) to monitor a log file or stdout and push new lines to CloudWatch. The agent watches the file, batches new lines, and sends them to the appropriate log group and stream.

For Python, the typical flow is:

1. Your Python app writes logs to stdout or to a file.
2. The CloudWatch agent (or a client library) reads those logs.
3. The agent sends log events to CloudWatch Logs via the AWS API.

You can also use the awslogs Python package to send logs directly, but the agent is the most common and robust approach for production workloads.

How it works step by step

Let’s break down the process of collecting Python logs with CloudWatch Logs:

Step 1: Configure Python logging

First, ensure your Python app logs in a structured format. Use the built-in logging module and set a formatter that includes timestamps and log levels:

import logging
import sys

logger = logging.getLogger("myapp")
logger.setLevel(logging.INFO)

handler = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter("%(asctime)s %(levelname)s %(name)s: %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)

This outputs logs to stdout, which is the easiest source for the agent to capture.

Step 2: Install the CloudWatch agent on your EC2 instance

On an EC2 instance running Amazon Linux 2 or Ubuntu, install and configure the CloudWatch agent. The agent can monitor log files or stdout.

Step 3: Create a log group and log stream in CloudWatch

You can create these via the AWS console or CLI. For example:

aws logs create-log-group --log-group-name /myapp/backend
aws logs create-log-stream --log-group-name /myapp/backend --log-stream-name ec2-instance-1

If you omit the stream, the agent can create it automatically.

Step 4: Configure the agent to watch your log source

Create a config file at /opt/aws/amazon-cloudwatch-agent/bin/config.json:

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/myapp.log",
            "log_group_name": "/myapp/backend",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

Here, {instance_id} is a placeholder that the agent replaces with the actual EC2 instance ID.

Step 5: Start the agent

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

Step 6: Run your Python app and watch logs appear

After the agent starts, new log lines from your app will appear in CloudWatch Logs within seconds.

That’s the core flow. But for development and testing, you might want a simpler, more direct way — which we’ll cover in the hands-on section.

Hands-on walkthrough

Let’s get your hands dirty. We’ll set up a minimal Python script, install the boto3 library, and write two approaches to collect logs. We’ll start with a local test that sends logs directly, then move to the agent-based approach.

Prerequisites

• An AWS account with appropriate permissions (CloudWatch Logs and EC2 if using the agent).
• Python 3.8+ installed locally.
• AWS CLI configured (or use boto3 with credentials).

Approach 1: Send logs directly with boto3 (for quick testing)

import boto3
import datetime

client = boto3.client("logs", region_name="us-east-1")

log_group = "/demo/python-app"
log_stream = "local-machine"

# Ensure log group and stream exist
client.create_log_group(logGroupName=log_group)
client.create_log_stream(logGroupName=log_group, logStreamName=log_stream)

# Build a log event (timestamp in milliseconds)
log_event = {
    "timestamp": int(datetime.datetime.now().timestamp() * 1000),
    "message": "Hello from Python!"
}

response = client.put_log_events(
    logGroupName=log_group,
    logStreamName=log_stream,
    logEvents=[log_event]
)

print("Log sent. Next sequence token:", response.get("nextSequenceToken"))

Expected output:

Log sent. Next sequence token: 495948443137227893833645999999999999999999999999

Note: For put_log_events in a sequence, you must use the sequenceToken from the previous response. However, the agent handles this automatically — for production, use the agent.

Approach 2: Use the CloudWatch agent to monitor a Python log file

1. Create a Python script that writes to a file:

import logging
import time

logger = logging.getLogger("file-app")
logger.setLevel(logging.INFO)

file_handler = logging.FileHandler("/var/log/myapp.log")
formatter = logging.Formatter("%(asctime)s %(levelname)s %(message)s")
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)

for i in range(5):
    logger.info("Request #%d processed", i)
    time.sleep(1)

1. Install and configure the agent on the EC2 instance as described earlier.

2. Run the script and then check CloudWatch Logs in the console or via CLI:

aws logs get-log-events --log-group-name /myapp/backend --log-stream-name ec2-instance-1

You’ll see the five log events appear.

Key takeaway from the walkthrough

The agent is the production-grade solution — it handles batching, retries, and multiple file sources. The direct boto3 method is useful for one-off scripts or for custom ingestion, but you’ll re-invent the wheel if you use it for every app.

Compare options / when to choose what

When you need to collect Python logs with CloudWatch Logs, you have several options. Here’s a comparison table:

When to choose what:

• For EC2 instances → Use the CloudWatch agent. It’s the standard and most reliable.
• For containerized apps on ECS → Use FireLens (or the agent in the container).
• For quick tests or Lambda → Use the awslogs library or boto3 directly.
• For serverless Lambda → You don’t need any setup; Python logs automatically go to CloudWatch.

Troubleshooting & edge cases

Even with a clear plan, things go wrong. Here are common issues and fixes:

Logs not appearing in CloudWatch

• Agent isn’t running. Check the agent status: sudo systemctl status amazon-cloudwatch-agent. Restart if needed.
• Wrong file path. The agent only reads the exact path you configured. Test with tail -f /var/log/myapp.log on the instance.
• Permissions. The agent needs permission to call logs:PutLogEvents. Ensure the IAM role attached to the instance has that policy.

Duplicate log events

• Agent and direct boto3 calls — If you use both methods for the same log stream, you’ll see duplicates. Stick to one method per stream.

Sequence token errors when using boto3

• Error: InvalidSequenceTokenException. This happens when you call put_log_events without a token. Use the token from the previous response, and handle the first call carefully (no token needed for the very first event).

Log group or stream missing

• The agent can create the log group if you set "force_flush_interval" and configure the log_group_name correctly. But if it doesn’t, create it manually via CLI before starting the agent.

Timeouts / slow ingestion

• The agent batches logs by default (up to 5 seconds). For real-time needs, reduce batch_interval or use the timeout setting. Check the agent logs at /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log.

What you learned & what's next

You now know how to collect Python logs with CloudWatch Logs using two main methods: the CloudWatch agent for production EC2 instances, and boto3 for direct, one-off sends. You understand the mental model of log groups, streams, and events, and you can choose the right tool for your scenario. You also know how to troubleshoot the most common issues — missing logs, permissions, and sequence token errors.

This skill is foundational for observability in AWS. With logs flowing into CloudWatch, you can next move on to CloudWatch Logs Insights to query and analyze your logs, or set up Metric Filters to turn log patterns into metrics and alarms. That’s a natural next step in your AWS Cloud & DevOps with Python journey — making your logs not just stored, but actionable.