Terraform Remote State with S3 Backend

Developers often start with Terraform using local state—a single terraform.tfstate file sitting in a directory. That works for solo experiments, but the moment you share infrastructure with a team, run CI/CD pipelines, or work from multiple machines, local state becomes a source of conflicts, overwritten changes, and security leaks. In this lesson, you'll learn how to use Terraform remote state with S3 backend to store your state safely in the cloud, enable locking, and build a foundation for collaboration and automation.

The problem this lesson solves

When you run terraform apply without a backend configuration, Terraform writes the state file locally. This creates several issues:

• Team collaboration: If two people run terraform apply at the same time, they can overwrite each other's state, leading to inconsistent infrastructure.
• CI/CD integration: In a pipeline, every runner or container needs access to the same state file. With local state, you'd have to copy the file around—which is fragile.
• Security: Local state files often contain sensitive values like database passwords or API keys. Storing them insecurely on a laptop or a shared drive is a risk.
• Disaster recovery: If your machine crashes, you can lose the entire state of your infrastructure.

Remote state solves this by centralizing the state storage. Using an S3 backend is the standard AWS approach, offering durability, versioning, and encryption. Combined with DynamoDB for state locking, you get a robust setup that supports concurrent operations and prevents corruption. This lesson walks you through the entire process, from creating the necessary resources to configuring Terraform to use them.

Core concept / mental model

Think of Terraform state as the source of truth for your infrastructure. It maps the resources you defined in code to real objects in AWS. When you run terraform plan, Terraform reads this state to understand the current situation; when you run apply, it updates the state to reflect changes.

Local state is like keeping a paper map on your desk—only you can see it. Remote state with S3 is like storing that map in a shared library where anyone with the right access can view and edit it. The S3 bucket is the library, and a special file (the state) is your map. Locking via DynamoDB is like a librarian tagging the map as "in use"—only one person can edit it at a time, preventing accidental conflicts.

The S3 backend isn't just a place to store the file. It also:

• Enables versioning of the state file, so you can roll back to a previous version if needed.
• Supports encryption at rest, protecting sensitive data.
• Works with IAM policies to control who can read/write the state.
• Works with DynamoDB to provide a lock, preventing concurrent modifications.

A mental model: backend "s3" tells Terraform to treat S3 as the storage layer, and the optional dynamodb_table adds a lock mechanism.

How it works step by step

The process of setting up an S3 backend follows a clear sequence:

1. Prepare the backend infrastructure: You need an S3 bucket (optionally with versioning enabled) and—if you want locking—a DynamoDB table. This is often done separately from your main Terraform configuration, sometimes in a bootstrap Terraform project or manually via the AWS CLI/console.
2. Write the Terraform configuration: In your main project, add a backend block inside the terraform block, specifying s3 as the backend, along with the bucket name, key (path to the state file), region, and optional DynamoDB table name.
3. Initialize Terraform: Run terraform init with the backend configuration. This sets up the backend and, if needed, prompts you to copy existing local state to the remote location.
4. Run plan/apply as usual: Terraform now uses the S3 bucket for state operations. Locking is automatically engaged when DynamoDB is configured.
5. For teams: Everyone using the same configuration will share the same state. Ensure IAM permissions are set appropriately.

It's important to note that the S3 bucket itself must exist before you run terraform init. You cannot create the bucket and use it as the backend in the same configuration run, because the backend is initialized before any resources are created.

Hands-on walkthrough

Let's build a real example. We'll first create the S3 bucket and DynamoDB table using the AWS CLI or a separate Terraform config, then use those resources as the backend.

Step 1: Create the S3 bucket and DynamoDB table

Using the AWS CLI (ensure you have configured credentials), run:

# Create the bucket (name must be globally unique)
aws s3api create-bucket --bucket my-terraform-state-bucket --region us-east-1

# Enable versioning
aws s3api put-bucket-versioning --bucket my-terraform-state-bucket --versioning-configuration Status=Enabled

# Create the DynamoDB table for locking
aws dynamodb create-table --table-name terraform-lock --attribute-definitions AttributeName=LockID,AttributeType=S --key-schema AttributeName=LockID,KeyType=HASH --billing-mode PAY_PER_REQUEST

Expected output: A JSON response confirming the bucket creation, and a table status of ACTIVE.

Step 2: Configure Terraform with S3 backend

Create a file main.tf in your project:

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "dev/network/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-lock"
  }
}

# Example resource: an S3 bucket you want to manage
resource "aws_s3_bucket" "example" {
  bucket = "my-app-static-assets"
}

Run terraform init in the project directory. Terraform will respond with a message like "Initializing the backend..." and ask to copy existing state if there's a local terraform.tfstate file. Enter yes to migrate.

Now, every terraform plan and terraform apply reads from S3 and locks using DynamoDB.

Step 3: Use a Python script to verify the state

Since this track includes Python, let's write a simple script to list the versions of the state file in S3, demonstrating that the remote state is being used.

import boto3

bucket_name = "my-terraform-state-bucket"
key = "dev/network/terraform.tfstate"

s3 = boto3.client("s3")
response = s3.list_object_versions(Bucket=bucket_name, Prefix=key)

print("State file versions:")
for version in response.get("Versions", []):
    print(f"  ID: {version['VersionId']}  LastModified: {version['LastModified']}")

Run it with python check_state.py and you should see at least one version listed after your first terraform apply. This confirms the state is safely stored and versioned in S3.

Compare options / when to choose what

While S3 is the recording artist for this lesson, there are other remote backends. Let's compare the most common:

When to choose what: - S3 backend is the default choice for AWS-only teams because it's simple, secure, and integrates with IAM and KMS. It's ideal for most production setups. - Terraform Cloud is great for teams that want a managed solution with built-in remote operations, policy, and audit logs—but it introduces a third-party dependency. - Other cloud providers only make sense if you already operate there for other resources.

For this track, focus on S3 + DynamoDB. It's the most versatile and commonly used in AWS DevOps environments.

Troubleshooting & edge cases

Here are common pitfalls and how to handle them:

"Backend initialization failed"

Error: Error: Failed to get existing workspaces: AccessDenied: Access Denied

Cause: The IAM user or role doesn't have permission to read or write to the S3 bucket.

Fix: Attach an IAM policy like AmazonS3FullAccess (scoped down for production) and ensure the DynamoDB table has GetItem, PutItem, DeleteItem permissions.

"Error acquiring the state lock"

Cause: Another terraform process holds the lock, or a previous crash left the lock in place.

Fix: Wait for the other process to finish, or if it’s stale, use terraform force-unlock <lock_id> (use with caution).

"State file not found"

Error: Error: no existing Terraform state found

Cause: The key path points to a nonexistent object. If you just configured the backend, the state won't be there until you run terraform apply.

Fix: Run terraform apply to create the state file. If you have local state, migrate it via terraform init.

"Versioning not enabled"

If you want to protect against accidental corruption, enable S3 versioning. Without it, you can't roll back to a previous state. Enable it immediately after bucket creation.

"Using the bucket for both state and resources"

Avoid using the same S3 bucket for state files and application data. It complicates IAM policies and can lead to accidental deletion of state.

What you learned & what's next

You now understand how to use Terraform remote state with S3 backend—from creating the bucket and DynamoDB table to configuring the backend and running Terraform commands. You learned how to enable locking and versioning, and how to troubleshoot common issues. This is a cornerstone of team-based IaC and CI/CD workflows.

As a next step, consider learning about Terraform workspaces or Terraform Cloud to manage environments like dev, staging, and production. In the next lesson, you'll likely explore Terraform modules to encapsulate reusable infrastructure patterns—another layer of professionalism in your DevOps toolkit.