Set Up VPC Flow Logs

Your VPC is humming along, workloads are passing traffic, and then — one day — something feels off. An unusual API call, a spike in outbound traffic, a connection to an IP you don't recognize. Without visibility into the network flows inside your VPC, you're flying blind. You could spend hours tracing through logs, or you could set up VPC flow logs and get a clear, queryable record of every connection attempt happening inside your cloud environment. This lesson will give you the exact steps to turn on that visibility and make it a cornerstone of your cloud security discovery process.

The problem this lesson solves

Cloud security isn't just about preventing attacks; it's about detecting them early and understanding their footprint. Traditional security tools monitor host-level activity, but they miss the network layer. A compromised container, a misconfigured security group, or an attacker pivoting through your VPC — none of these leave obvious traces on disk. They do leave network traces.

Without VPC flow logs, you have no answer to questions like: - Who is connecting to my database server on port 5432? - Why is my web server sending traffic to an IP in a sanctioned region? - Which security group rule is allowing unexpected traffic?

Enabling VPC flow logs gives you a per-connection record — source, destination, port, protocol, and whether the connection was accepted or rejected. This is your first line of discovery for anything that moves across your network.

Pro tip: Flow logs are not a security control that blocks malicious traffic. They are a detection mechanism — they log, they don't block. Pair them with network ACLs and security groups for actual enforcement.

Core concept / mental model

Think of VPC flow logs as a flight recorder for your VPC. Like the black box on an airplane, it continuously records what's happening in the network — every takeoff (connection start) and landing (connection end) — without interfering with the flight itself.

When you enable flow logs for a VPC, subnet, or network interface, the service captures metadata about each traffic flow: source IP, destination IP, source port, destination port, protocol, and actions (ACCEPT or REJECT). The logs are published to a destination — typically Amazon S3 or CloudWatch Logs — and you can query them later with tools like Athena or CloudWatch Logs Insights.

Key definitions

• Flow log: A record of network traffic metadata (not payload).
• Traffic flow: A unidirectional stream of IP packets that share source, destination, and port.
• Aggregation interval: The window (1 minute or 10 minutes) over which flows are captured and logged.
• Capture window: The period between when data is captured and when it appears in the destination — usually a few minutes.

Visualizing the flow

VPC
 ├─ Subnet A (public)
 │    └─ EC2 instance (web server)
 │         └─ ENI (elastic network interface) — flow logs attached
 ├─ Subnet B (private)
 │    └─ RDS database — flow logs attached at VPC level
 └─ Flow logs → S3 bucket → Athena queries

This mental model scales: you start with a single VPC, then extend flow logging to all VPCs in your organization as you mature.

How it works step by step

Setting up VPC flow logs for discovery follows a predictable sequence. Here's the high-level flow, and then we'll get into hands-on details.

1. Choose the scope: VPC, subnet, or a specific network interface. VPC-level captures everything inside that VPC.
2. Pick the destination: S3 bucket (best for long-term storage & analytics) or CloudWatch Logs (good for real-time monitoring).
3. Set the format: Choose the fields you want to include — full format includes all fields like srcaddr, dstaddr, srcport, dstport, protocol, action.
4. Enable the log through the AWS console, CLI, or Infrastructure as Code (like Terraform/CloudFormation).
5. Query your logs: Once logs are flowing, use Athena (on S3) or Logs Insights to search for anomalies.

Cause and effect

• Enable flow logs → AWS starts publishing log records every minute (if you choose 1-minute aggregation) into your S3 bucket.
• Missing IAM permissions → Log delivery fails silently — you'll see no logs and no error in the console unless you check the flow log's Status.
• Adding more fields → Bigger log records, slightly higher S3 costs, but richer data for security analysis.

Hands-on walkthrough

Let's get practical. I'll show you how to set up VPC flow logs for discovery using two approaches: the AWS CLI (for scriptable control) and Terraform (for infrastructure as code).

Prerequisites

• AWS account with an existing VPC.
• AWS CLI configured (aws configure) with sufficient permissions (ec2:CreateFlowLogs, s3:PutObject, etc.).
• Basic familiarity with JSON and command line.

Step 1: Create an S3 bucket for logs

aws s3 mb s3://my-vpc-flow-logs-bucket --region us-east-1

Step 2: Enable flow logs via AWS CLI

Get the VPC ID first:

aws ec2 describe-vpcs --region us-east-1

Then create the flow log:

aws ec2 create-flow-logs \
  --resource-type VPC \
  --resource-id vpc-0abc123def4567890 \
  --traffic-type ALL \
  --log-destination-type s3 \
  --log-destination arn:aws:s3:::my-vpc-flow-logs-bucket \
  --max-aggregation-interval 60 \
  --log-destination-format '{"FieldNames":["version","account-id","interface-id","srcaddr","dstaddr","srcport","dstport","protocol","packets","bytes","start","end","action","log-status"]}'

Expected output:

{
    "FlowLogIds": [
        "fl-0abc123def4567890"
    ]
}

Step 3: Verify logs are being delivered

List your flow logs:

aws ec2 describe-flow-logs --region us-east-1 --filter "Name=flow-log-id,Values=fl-0abc123def4567890"

Check the Status field — it should be ACTIVE. After a few minutes, query S3:

aws s3 ls s3://my-vpc-flow-logs-bucket/ --recursive | head

You should see files with a path like AWSLogs/<account-id>/vpcflowlogs/us-east-1/YYYY/MM/DD/.

Step 4: Query with Athena (optional but powerful)

Once logs are in S3, you can run SQL-like queries. Create a table in Athena pointing to the S3 prefix, then run:

SELECT srcaddr, dstaddr, dstport, action, COUNT(*) AS attempts
FROM vpc_flow_logs
WHERE dstport = 3306
GROUP BY srcaddr, dstaddr, dstport, action
ORDER BY attempts DESC;

This reveals all attempts to connect to MySQL (port 3306) and whether they were accepted or rejected.

Hands-on with Terraform

If you manage infrastructure as code, here's the Terraform equivalent:

resource "aws_vpc_flow_log" "example" {
  iam_role_arn    = aws_iam_role.flow_logs.arn
  log_destination = aws_s3_bucket.flow_logs.arn
  traffic_type    = "ALL"
  vpc_id          = aws_vpc.main.id
}

Compare options / when to choose what

When to choose what?

• Security discovery at scale → S3 + Athena (cheap, flexible).
• Responding to live incidents → CloudWatch Logs (fast search) + CloudWatch Alarms.
• Both? You can even send to both! AWS supports two destinations per flow log.

Variations

• Subnet-level flow logs: More granular than VPC-level — good for isolating noisy subnets.
• Custom fields: Include tcp-flags to detect connection open/close patterns — useful for port scanning detection.
• Third-party tools: Ship flow logs to tools like Splunk, Datadog, or Sumo Logic via S3/CloudWatch integrations.

Troubleshooting & edge cases

• No logs appearing after hours → Check the flow log status: aws ec2 describe-flow-logs and look for Status = ERROR. Common cause: bucket policy missing. Your S3 bucket must have a policy allowing aws-logs service to write. json { "Effect": "Allow", "Principal": {"Service": "delivery.logs.amazonaws.com"}, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::my-vpc-flow-logs-bucket/AWSLogs/*" }

• Logs are delayed by more than 5 minutes → Flow log capture is asynchronous; delays up to 10-15 minutes are normal. If it exceeds 15, check for network connectivity to the destination region.

• Empty srcaddr/dstaddr fields → These appear when the flow log captures traffic that doesn't have an IP (e.g., certain AWS-internal services). Filter them out when querying.

• Cost shock → Flow logs can generate large volumes. Use the 10-minute aggregation interval (instead of 1-minute) for less granular but cheaper capture, and set S3 lifecycle rules to expire old logs.

• Permissions issue when using CloudWatch → Ensure your IAM role has logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents. Also, the role trust policy must allow vpc-flow-logs.amazonaws.com.

What you learned & what's next

You now understand the core concept of VPC flow logs for discovery — what they are, how they work, and how to set them up using the CLI and Terraform. You learned how to choose between S3 and CloudWatch destinations, and you know how to troubleshoot common issues like missing bucket policies or silent permission failures.

You've met these learning objectives: - Explain the core idea behind setting up VPC flow logs for discovery. - Complete a practical exercise (CLI or Terraform) to enable flow logs in your environment.

Now that you have network visibility, you're ready for the next lesson in the Cloud security essentials track. You'll likely dive into analyzing these logs for threat hunting, or automating alerts based on suspicious traffic patterns. Stay sharp — the data is now flowing, but without proper analysis, it's just noise.

Happy hunting!