Centralize Logs in CloudWatch

You've built an amazing application and deployed it to AWS, the 'us-east-1' region. But when a security incident happens, you find yourself SSH-ing into each EC2 instance, digging through /var/log/application.log files by hand. Your logs are scattered across dozens of instances, some of which auto-scale and disappear, and you have no centralized view. This is the exact pain this lesson solves: centralizing logs in CloudWatch from your apps so that every log entry flows into one searchable, monitorable, and alertable place. Let's fix this today.

The problem this lesson solves

When your application runs on multiple EC2 instances, containers, or Lambda functions, each one writes its own local log files. Troubleshooting becomes a nightmare: you have to guess which instance saw the error, patch together log fragments, and you often miss the 'smoking gun' because the instance was terminated during a scale-in event.

Beyond the operational chaos, this lack of centralization is a security risk. Security teams need a single source of truth to spot brute-force attempts, API misuse, or data exfiltration patterns across your entire fleet. Manual log collection is slow, error-prone, and almost never real-time.

The solution is to stream all application logs to a central log management service—Amazon CloudWatch Logs. This gives you:

• Single pane of glass: Search across all instances, containers, and functions.
• Real-time monitoring: Set up alarms and dashboards on log patterns.
• Security and compliance: Preserve immutable logs for audits and forensics.

Core concept / mental model

Think of your application logs as water from many streams. Each app is a stream, and CloudWatch Logs is the reservoir. Instead of going to each stream to taste the water, you install pumps (agents) that continuously push water into the reservoir. Then you can sample, filter, and test the water all in one place.

In AWS terms, the key components are:

• Log group: The reservoir—a container for logs from a single application or service. E.g., /ec2/my-app, /aws/lambda/orders-function.
• Log stream: A stream within the group—usually one per EC2 instance, container, or Lambda invocation. E.g., i-0ab3c4d5e6, 2024/05/01/[$LATEST]c9e6.
• Log event: A single log entry—a timestamp and a message.
• CloudWatch agent: The pump—a small daemon on your EC2 instance that reads local log files and sends them to CloudWatch Logs.

This architecture scales: one agent per instance, hundreds of streams per group, and you query across them all in seconds.

How it works step by step

Here's the high-level flow of centralizing logs in CloudWatch from apps:

1. Prepare your application to write logs to a known location (e.g., /var/log/my-app/application.log) in a structured format (JSON or key-value).
2. Install the CloudWatch agent on each EC2 instance or container host. For Lambda, you don't need an agent—the runtime automatically sends logs to CloudWatch.
3. Configure the agent with a JSON config that specifies which log files to watch, the log group, stream name format, and timestamp format.
4. Start the agent and verify it's sending logs.
5. Query and monitor your logs in the CloudWatch console, set up metric filters, and create alarms.

For containerized apps, you can use the Fluentd/Fluent Bit plugins instead of the agent. For on-premises servers, the same agent works with a customer account.

Hands-on walkthrough

Let's get our hands dirty. We'll centralize logs from a simple Python web app on an EC2 instance to CloudWatch Logs.

1. Install the CloudWatch agent

First, connect to your EC2 instance (Amazon Linux 2) and install the agent:

sudo yum install -y amazon-cloudwatch-agent

2. Create a CloudWatch agent configuration

The agent config is a JSON file. Here's an example that collects logs from /var/log/my-app/application.log:

{
  "agent": {
    "metrics_collection_interval": 60
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/my-app/application.log",
            "log_group_name": "/ec2/my-app",
            "log_stream_name": "{instance_id}",
            "timestamp_format": "%Y-%m-%dT%H:%M:%S%z",
            "timezone": "UTC"
          }
        ]
      }
    }
  }
}

Save this as /opt/aws/amazon-cloudwatch-agent/bin/config.json on the instance.

3. Start the agent

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
    -a fetch-config \
    -m ec2 \
    -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json \
    -s

This fetches and starts the agent. Now let's generate some logs with a quick Python script and verify they arrive in CloudWatch.

4. Generate and verify logs

Create a simple Python app that writes a log with a unique message:

import logging
import time

logging.basicConfig(
    filename="/var/log/my-app/application.log",
    level=logging.INFO,
    format="%(asctime)s %(message)s",
    datefmt="%Y-%m-%dT%H:%M:%S%z"
)

for i in range(5):
    logging.info(f"Order processed: order_{i}")
    time.sleep(1)

Run it: python3 app.py. Then in the AWS console, go to CloudWatch > Log groups > /ec2/my-app and see your log streams. You should see the logs like:

2024-05-01T15:30:01+0000 Order processed: order_0
2024-05-01T15:30:02+0000 Order processed: order_1
...

Compare options / when to choose what

There are several ways to centralize logs in CloudWatch, and your choice depends on your environment:

When to choose: For classic EC2 fleets, the agent is the go-to. For serverless, Lambda logging is automatic. For Kubernetes, use Fluent Bit. For quick prototyping, you can use the API directly.

Troubleshooting & edge cases

• No logs appear: Check the agent status with sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status. Also verify the log file path exists and has read permissions for the cwagent user.
• Incorrect timestamps: If logs show wrong times, set the timestamp_format to match your log format. You can use %Y-%m-%d %H:%M:%S or JSON logs with a timestamp key.
• Logs truncated: The agent has a max log size per event (256 KB). If your messages are larger, consider splitting them or using the API.
• IAM permissions: The instance profile must have logs:PutLogEvents, logs:CreateLogStream, and logs:CreateLogGroup. Missing permissions cause silent failures.
• Multi-line logs: By default, the agent treats each line as a separate event. For stack traces, set multi_line_start_pattern to match the first line of your stack trace.

What you learned & what's next

You've learned the core concepts of centralizing logs in CloudWatch from apps: the problem of scattered logs, the mental model of log groups and streams, the step-by-step setup with the CloudWatch agent, and how to troubleshoot common issues. You now know how to stream logs from EC2 instances, and you understand alternative approaches for Lambda and containers.

Next step: In the next lesson, you'll build on this by setting up alarms and notifications on your centralized logs—so you're not just collecting logs, but actively responding to security events. That's where log-based monitoring turns into proactive defense.