EC2 User Data for Python Apps

Manually SSHing into every EC2 instance to install Python, copy your app, and start a process is a tedious, error-prone ritual. It's slow, impossible to scale, and a breeding ground for configuration drift. This lesson introduces EC2 user data — a simple, powerful feature that automates the entire bootstrap process, letting you launch a fully configured server with a single API call.

The problem this lesson solves

When you launch an Amazon EC2 instance, you get a bare, generic virtual machine. It has an operating system, but no Python, no application code, and no running services. Without an automated bootstrap, every instance requires manual steps:

1. SSH into the instance.
2. Install Python and required system packages.
3. Copy your application code (via scp, git clone, etc.).
4. Install Python dependencies (pip install -r requirements.txt).
5. Start your application with nohup, systemd, or a similar tool.
6. Repeat for every instance in your fleet.

This workflow breaks down fast. It’s time-consuming, difficult to repeat consistently, and nearly impossible to automate properly. If you ever need to scale from 1 instance to 10, you’ll find yourself repeating the same steps over and over, hoping you didn’t miss a library or a config file.

Moreover, manual processes are a major source of configuration drift — each instance might have slightly different versions of packages or overlook a critical step. EC2 user data solves this by letting you specify a script that runs automatically during the instance’s first boot, turning a raw VM into a production-ready Python host.

Core concept / mental model

Think of an EC2 instance as a new smartphone and user data as the setup wizard that runs when you first power it on. The wizard asks you a few things (like Wi-Fi password, accounts, preferences), then configures the phone exactly as you want — all before you even see the home screen.

Similarly, EC2 user data is a script you provide when launching an instance. AWS runs that script on first boot, automatically installing packages, cloning code, and starting services — no SSH needed. The key idea is declarative bootstrapping: you declare what the final state should be, and the system makes it happen.

Here’s how it fits into the broader picture:

• User data is not an AWS-specific language — it’s just a shell script (or a cloud-init directive) that runs as root on the instance.
• It runs once, on the very first boot. On subsequent reboots, it does not rerun unless the instance is re-launched with fresh storage.
• User data is stored in the instance metadata and is visible to the instance itself; it’s not meant for long-term secrets (see Troubleshooting).

Cloud-init, the underlying tool (on Amazon Linux and Ubuntu), processes the user data and supports multiple formats: shell scripts (#!/bin/bash), cloud-config YAML, and more. For Python apps, you’ll most often use a shell script that sets up the environment and launches the service.

How it works step by step

Here’s the precise sequence when you launch an instance with user data:

1. You specify the user data when you run run_instances (via the AWS Console, CLI, or SDK). This is a text blob, often a shell script.
2. AWS places the instance in the “pending” state. The instance is allocated and starts booting; the underlying hypervisor prepares the virtual hardware.
3. The OS boots. The kernel starts, init system (systemd) comes up, and network interfaces are configured. If you’re using a VPC, the instance gets an IP and metadata access is available.
4. Cloud-init reads the user data. On first boot, cloud-init fetches the user data from the instance metadata service (at http://169.254.169.254/latest/user-data) and executes it.
5. Your script runs as root. Any command you include — installing packages, cloning repos, writing files — runs with full privileges. Output is logged (typically to /var/log/cloud-init-output.log on Amazon Linux or Ubuntu).
6. Your application starts. You can launch your Python app directly in the script (with nohup or a systemd service) and it will keep running after the script finishes.

Crucially, the script runs as part of the boot process — so if it fails (e.g., a package name is wrong), the instance may become unusable for your application. That’s why testing in a controlled environment is wise before rolling out to production.

Hands-on walkthrough

Let’s get practical. You’ll launch an EC2 instance that automatically installs Python 3, clones a simple Flask app, installs its dependencies, and starts a systemd service — all via user data.

Prerequisites

• An AWS account and credentials configured for the AWS CLI.
• A key pair (if you want to SSH later for verification, though it’s not required).
• A security group that allows inbound HTTP on port 8000 (optional but useful).

Step 1: Write your user data script

Create a file called bootstrap.sh with the following content:

#!/bin/bash
set -e  # Exit on any error

echo "=== Starting user data script ==="

# Update package lists and install Python 3, pip, git
apt-get update -y
apt-get install -y python3 python3-pip git

# Create a directory for the app
mkdir -p /opt/myapp
cd /opt/myapp

# Clone a simple Flask app (replace with your own repo)
git clone https://github.com/your-username/simple-flask-app.git .

# Install Python dependencies
pip3 install -r requirements.txt

# Create a systemd service to run the app automatically
cat > /etc/systemd/system/myapp.service <<'EOF'
[Unit]
Description=My Python App
After=network.target

[Service]
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/python3 app.py
Restart=always
User=root

[Install]
WantedBy=multi-user.target
EOF

# Enable and start the service
systemctl enable myapp.service
systemctl start myapp.service

echo "=== Bootstrap complete ==="

Pro tip: Use set -e at the top of your script. If any command fails, the script stops and you can investigate the logs instead of letting a broken state persist.

Pro tip: Replace your-username/simple-flask-app.git with your own repo. For learning, you can use a public repo like https://github.com/realpython/flask-hello-world.git (adjust the app file if needed).

Step 2: Launch an instance with user data via the AWS CLI

Use the AWS CLI to launch a t2.micro instance (free tier eligible) with your script:

aws ec2 run-instances \
    --image-id ami-0abcdef1234567890 \
    --instance-type t2.micro \
    --key-name my-key-pair \
    --security-group-ids sg-0123456789abcdef0 \
    --subnet-id subnet-0123456789abcdef0 \
    --user-data file://bootstrap.sh

Expected output: A JSON response with the new instance ID and state pending. The instance will boot, and your script will run automatically.

Step 3: Verify the app is running

Wait a minute or two, then get the public IP:

aws ec2 describe-instances --instance-ids i-1234567890abcdef0 \
  --query 'Reservations[0].Instances[0].PublicIpAddress' --output text

Then, browse to http://<public-ip>:8000/ (if your security group allows port 8000). You should see your Flask app’s homepage. No SSH, no manual setup.

Step 4: Check the logs (if something goes wrong)

If the app isn’t responding, SSH into the instance and inspect the cloud-init logs:

ssh -i my-key.pem ec2-user@<public-ip>
sudo cat /var/log/cloud-init-output.log

You’ll see the output of your script, including any error messages.

Compare options / when to choose what

User data is just one of several ways to bootstrap EC2 instances. Here’s how it stacks up against common alternatives:

When to choose user data:

• Learning and prototyping — quickest way to get a working instance.
• Simple, one-time setup — e.g., a dev server or a single test instance.
• As a stepping stone — you’ll often embed user data inside Terraform or CloudFormation later.

When NOT to choose user data:

• Large fleets — launching 100 instances at once with a heavy install script can be slow and cause a thundering herd (all instances hitting package repos simultaneously). Pre-built AMIs are faster.
• Frequent updates — if you change your script, you must relaunch instances. Config management or AMIs handle updates more gracefully.
• Strict compliance or auditing — user data is not versioned; it’s hard to track who changed what.

Troubleshooting & edge cases

Even with a perfect script, things can go wrong. Here are the most common issues and how to diagnose them:

Issue: The script ran but the app isn’t running

• Check the log: SSH in and view /var/log/cloud-init-output.log. Look for errors near the end.
• Check the service: Run sudo systemctl status myapp to see if it’s active and if it crashed.
• Check the port: Maybe the app is listening on a different port (e.g., 5000 instead of 8000). Use sudo netstat -tulpn | grep python to see actual listening ports.

Issue: User data didn’t run at all

• Confirm it’s first boot: If the instance was launched from a snapshot that had already booted once, user data may not run again. Relaunch from a fresh AMI.
• Check the script format: The first line must be a shebang (#!/bin/bash) or start with #cloud-config for cloud-init to recognize it.
• Check permissions: The script doesn’t need to be executable, but if you have a syntax error, cloud-init will log it. Look at /var/log/cloud-init.log.

Issue: apt-get update fails due to network or permission

• Ensure your security group allows outbound HTTPS (port 443) and HTTP (port 80). Many default security groups allow all outbound traffic, but if you restricted it, this will fail.
• If you’re using Amazon Linux, use yum instead of apt-get. The script syntax differs.

Issue: Secrets in user data

Never hardcode passwords or API keys in user data. It’s stored in plaintext in the instance metadata and visible to anyone with instance access (or via the console if you’re not careful).

Better approach: Use IAM roles and the AWS Secrets Manager or Parameter Store to retrieve secrets at runtime.

Issue: Script runs every reboot (unexpectedly)

By default, user data only runs on first boot. But if you’re using cloud-init with certain modules (like runcmd), it might run on every boot. To force one-time execution, make sure your script is idempotent — it should be safe to run multiple times without side effects.

What you learned & what's next

You now understand how to use EC2 user data to bootstrap Python apps. You’ve seen the problem of manual setup, the mental model of first-boot scripts, a step-by-step walkthrough with a real Flask app, and how to compare user data with other approaches like pre-built AMIs and configuration management. You also know how to troubleshoot common failures and avoid security pitfalls.

Key takeaways to remember:

• User data is a first-boot script that automates installation, code deployment, and service startup.
• Cloud-init handles the execution and logs to /var/log/cloud-init-output.log.
• User data is perfect for simple, single-instance setups but not for large, dynamic fleets.
• Always test your script before deploying to production.

What’s next? In the next lesson, you’ll learn how to combine EC2 user data with infrastructure as code — you’ll put user data into a Terraform or CloudFormation template so you can launch your Python app with a single command, making your setup fully repeatable and version-controlled. This is the foundation for building robust CI/CD pipelines that deploy your app to EC2 automatically.