Python with Terraform Workflows

Every DevOps engineer who has spent a Friday afternoon chasing a manually-applied terraform apply that drifted from the codebase knows the pain. You’re not just typing commands — you’re orchestrating states, managing outputs, and hoping nothing breaks between plan and apply. The solution is to use Python with Terraform workflows, wrapping the Terraform CLI in scripts that are repeatable, testable, and deterministic. This lesson hands you the battle-tested pattern for turning infrastructure chaos into a scripted, observable pipeline.

The problem this lesson solves

Terraform is a fantastic declarative language for defining infrastructure, but it has a dirty secret: it’s not a general-purpose programming language. When you need to loop over complex logic, read environment-specific values from a CSV, or conditionally apply different configurations based on a CI/CD matrix, HCL alone becomes painful. You end up with duplicated terraform init commands scattered across shell scripts, fragile terraform output parsing with grep, and no way to unit-test your deployment logic.

Consider a typical enterprise: three environments (dev, staging, prod), each with separate variable files and state backends. Manually running terraform apply -var-file="dev.tfvars" in each directory works — until someone forgets to run terraform validate first, or the backend bucket isn’t initialized. Python fills that gap. You use Python with Terraform workflows to build a thin, robust orchestration layer that:

• Manages the order of Terraform commands (init, plan, apply, destroy).
• Parses and validates variables before they hit Terraform.
• Captures and inspects outputs for post-deployment checks.
• Integrates cleanly with your existing Python-based DevOps tools (e.g., boto3, pytest).

The rule of thumb: Terraform declares what the infrastructure should be; Python decides when and how to apply that declaration.

Core concept / mental model

Think of Python as the conductor of an orchestra, and Terraform as the sheet music. The sheet music (.tf files) defines every note, every instrument, and the exact desired melody (infrastructure state). The conductor (Python script) walks on stage, tells each musician when to start, checks their tuning, and ensures the performance doesn’t go off-key. Without the conductor, you have chaos — musicians playing at random times, or worse, playing the wrong version of the song.

More technically, the mental model has three layers:

• The Terraform layer: terraform init, plan, apply, destroy — low-level, stateful, and idempotent when used correctly.
• The Python orchestration layer: subprocess or a library like python-terraform that wraps those commands, handles exit codes, and parses JSON outputs.
• The automation layer: CI/CD pipelines, event triggers, or scheduled jobs that call your Python scripts. This is where the real value lies — you can now retry, log, and alert on every infrastructure change.

Key definitions

• Subprocess: Python’s module to spawn new processes and interact with their input/output streams — your primary bridge to the terraform binary.
• terraform output -json: A structured way to get Terraform state values as JSON, which Python can parse natively.
• Exit code 0: The universal signal that a command succeeded; anything else means failure and needs handling.

How it works step by step

Now let’s map the step-by-step flow that any Python-driven Terraform workflow follows. You’ll see this pattern again and again in real-world DevOps code.

1. Setup and environment check: Verify that Terraform is installed and that your working directory contains a main.tf. Python can check with shutil.which("terraform") and os.path.exists().
2. Initialize: Run terraform init to download providers and modules. This must happen before any other command. In a CI environment, you often do this once per machine, but in a scripted workflow you always call it to ensure reproducibility.
3. Plan: Execute terraform plan -out=tfplan to generate a plan file. Critical: -out=tfplan saves the plan to a binary file, which you can later apply. This prevents the infamous “plan drift” where someone changes infrastructure between planning and applying.
4. Validate (optional but recommended): Parse the plan’s JSON output (terraform plan -json) to check for unexpected changes. Python can compare against a whitelist of allowed changes (e.g., allow tags updates but fail on instance type changes).
5. Apply: Run terraform apply tfplan. Capture the output for logs. Use -auto-approve only in non-interactive environments (like CI), and always after a successful plan.
6. Extract outputs: After apply, run terraform output -json and load the result into a dictionary. This gives you resource identifiers (like an EC2 instance ID) that your Python code can pass to other tools.
7. Post-deployment checks: In Python, assert that outputs match expected patterns (e.g., IP address is not empty). If a check fails, you can trigger a rollback (e.g., call terraform destroy).

Why this order?

Each step depends on the previous one’s outputs. init creates the local state and module cache; plan reads the current state and compares with desired; apply consumes the plan file. Skipping steps leads to errors — as you’ll see in troubleshooting.

Hands-on walkthrough

Let’s get your hands dirty. We’ll write a Python script that runs init, plan, and apply in a controlled way, while capturing outputs for post-deployment verification.

Prerequisites

• Python 3.10+ installed.
• Terraform binary in your PATH (test with terraform version).
• A simple Terraform config. Create a directory tf-project with a main.tf like this:

# tf-project/main.tf
provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "example" {
  bucket_prefix = "my-app-data"
}

output "bucket_id" {
  value = aws_s3_bucket.example.id
}

Script 1: Basic subprocess orchestration

Create a file terraform_orchestrator.py in the parent directory.

import subprocess
import sys
import json

TERRAFORM_DIR = "tf-project"

def run_terraform(command, args=None, check=True):
    """Run terraform with given command and args, returning completed process."""
    cmd = ["terraform", command]
    if args:
        cmd.extend(args)
    print(f"Running: {' '.join(cmd)}")
    result = subprocess.run(cmd, cwd=TERRAFORM_DIR, capture_output=True, text=True)
    if result.returncode != 0 and check:
        print(f"Terraform {command} failed:\n{result.stderr}")
        sys.exit(result.returncode)
    return result

def main():
    # Step 1: init
    run_terraform("init")

    # Step 2: plan (with output to a plan file)
    plan_file = "tfplan"
    run_terraform("plan", ["-out=" + plan_file])

    # Step 3: apply the saved plan
    run_terraform("apply", [plan_file])

    # Step 4: get outputs as JSON
    output_res = run_terraform("output", ["-json"])
    outputs = json.loads(output_res.stdout)

    # Post-deployment check
    bucket_id = outputs["bucket_id"]["value"]
    print(f"Bucket created: {bucket_id}")
    assert bucket_id.startswith("my-app-data"), "Unexpected bucket name!"
    print("All checks passed.")

if __name__ == "__main__":
    main()

Run it:

python terraform_orchestrator.py

Expected output (truncated for brevity):

Running: terraform init
Terraform has been successfully initialized!
Running: terraform plan -out=tfplan
...
Plan: 1 to add, 0 to change, 0 to destroy.
Running: terraform apply tfplan
...
Outputs:

bucket_id = "my-app-data202501011234"
Bucket created: my-app-data202501011234
All checks passed.

Pro tip: Always use -out=tfplan when running plan programmatically. It ensures you apply exactly what you planned, avoiding the “Hey, I didn’t create that instance!” blame game.

Script 2: Using environment-specific variables

Now let’s make it configurable. Create a tf-vars directory with dev.tfvars and prod.tfvars.

dev.tfvars:

bucket_prefix = "dev-app"
environment    = "dev"

prod.tfvars:

bucket_prefix = "prod-app"
environment    = "prod"

Update main.tf to accept variables:

variable "bucket_prefix" {}
variable "environment" {}

resource "aws_s3_bucket" "example" {
  bucket_prefix = var.bucket_prefix
}

output "bucket_id" {
  value = aws_s3_bucket.example.id
}

output "environment" {
  value = var.environment
}

Then modify the orchestrator to accept an environment argument:

import sys
import subprocess

TERRAFORM_DIR = "tf-project"
VAR_DIR = "tf-vars"

def run_terraform(command, args=None):
    cmd = ["terraform", command]
    if args:
        cmd.extend(args)
    result = subprocess.run(cmd, cwd=TERRAFORM_DIR, capture_output=True, text=True)
    if result.returncode != 0:
        print(f"Error: {result.stderr}")
        sys.exit(result.returncode)
    return result

def main(environment):
    var_file = f"{VAR_DIR}/{environment}.tfvars"
    # init (no backend config here, but in real life you'd pass -backend-config)
    run_terraform("init")

    # plan with variable file
    run_terraform("plan", ["-var-file=" + var_file, "-out=tfplan"])

    # apply
    run_terraform("apply", ["tfplan"])

    # get outputs
    output_res = run_terraform("output", ["-json"])
    import json
    outputs = json.loads(output_res.stdout)
    print(f"Deployed to {outputs['environment']['value']} - bucket: {outputs['bucket_id']['value']}")

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python orchestrator.py [dev|prod]")
        sys.exit(1)
    main(sys.argv[1])

Run with python terraform_orchestrator.py dev. Output:

Deployed to dev - bucket: dev-app202501011234

Now you have a single script that deploys to any environment — no more copy-paste commands.

Script 3: Using the python-terraform library (bonus)

While subprocess is enough, the python-terraform library (pip install python-terraform) wraps the same calls with a friendlier API. Be aware it’s not as actively maintained, so proceed with caution in production.

from python_terraform import Terraform

tf = Terraform(working_dir='tf-project', variables={'bucket_prefix': 'lib-demo'})
return_code, stdout, stderr = tf.init()
if return_code != 0:
    print(f"Init failed: {stderr}")
    exit(return_code)

return_code, stdout, stderr = tf.plan(out='tfplan')
if return_code != 0:
    print(f"Plan failed: {stderr}")
    exit(return_code)

return_code, stdout, stderr = tf.apply(skip_plan=True, plan_file='tfplan')
if return_code != 0:
    print(f"Apply failed: {stderr}")
    exit(return_code)

return_code, stdout, stderr = tf.output(json=True)
print(stderr)

The library does the same thing under the hood, but many teams prefer explicit subprocess calls for easier debugging.

Compare options / when to choose what

When you decide to use Python with Terraform workflows, you’ll face a fork in the road: subprocess vs. a dedicated library. The table below breaks down the trade-offs.

When to choose what?

• If you’re building a one-off script for a migration, subprocess is your friend — it’s transparent and works immediately.
• If you’re building a feature that will be maintained by a team for years, invest in a thin wrapper that uses subprocess but adds logging, retries, and typed outputs. This gives you the best of both worlds.
• If you are already using python-terraform and it meets your needs, stick with it — but be prepared to switch to subprocess if you hit a bug.

Variations worth mentioning

• HashiCorp’s CDK for Terraform (CDKTF): Instead of shelling out to the CLI, you write infrastructure as code in Python itself. This blurs the line between orchestration and definition — you get loops and conditionals natively. But it’s a bigger paradigm shift and adds a dependency on Node.js.
• Terragrunt with Python hooks: Some teams pair Terragrunt (a Terraform wrapper) with Python scripts invoked as before_hook or after_hook. This keeps the orchestration in a dedicated tool while using Python for custom validation.
• Using pyinfra or Ansible with Terraform: You can trigger Terraform from configuration management tools, but that’s adding too many layers for most use cases.

Troubleshooting & edge cases

Even with a flawless Python script, the real world bites. Here are the common monsters you’ll slay.

terraform init fails but works from terminal

• Cause: Your Python script’s working directory is wrong. subprocess.run(cwd=...) points to a path where main.tf doesn’t exist.
• Fix: Print os.getcwd() and os.listdir(cwd) before running the command for quick diagnosis.

Plan output is not JSON

• Cause: You forgot -json flag in the plan command, or you’re trying to parse the human-readable output.
• Fix: Use terraform plan -json and parse with json.loads on stdout. Be aware that the JSON format outputs a stream of log lines, so you might need to filter only lines that contain "@level": "planned_change".

Exit code is 2 but no error shows

• Cause: subprocess.run captures stderr, but you’re not printing it. Always print result.stderr on non-zero return code.
• Fix: Add the print as shown in the examples. In CI, log to a file instead of console.

apply fails with “Saved plan is stale”

• Cause: You ran plan without -out=tfplan, or you modified the config after planning (e.g., variable values changed via environment).
• Fix: Always pass -out=tfplan and reuse that file. Never attempt to plan and apply without saving in between.

Invalid variable type errors from Python

• Cause: variables dictionary values are passed as strings, but Terraform expects a list or number.
• Fix: If you pass variables via -var, ensure Python casts them correctly (e.g., str(1) for numbers, json.dumps([1,2]) for lists). Alternatively, use a tfvars file to avoid type guessing.

Common mistakes summary

• Running terraform init multiple times unnecessarily — it’s idempotent, but adds seconds to every run. Only call it when the provider version changes or you switch backend.
• Ignoring the -input=false flag in non-interactive environments. If a variable is missing, Terraform will prompt for input and hang your script. Always pass -input=false and validate variables in Python first.
• Parsing terraform output without -json. The plain text output has trailing newlines and quotes, leading to brittle regex parsing. Use JSON and you’ll thank yourself later.
• Not capturing stderr in logs. The most informative errors (like AWS permission issues) appear there.
• Assuming terraform apply is reversible. Once you apply, there’s no undo — always have a plan for destroy, and test that Python can call terraform destroy safely.

What you learned & what's next

You now have the power to use Python with Terraform workflows — from running a simple init-plan-apply cycle to parameterizing environments and validating outputs. You learned:

• How to orchestrate Terraform CLI commands with Python’s subprocess.
• Why saving the plan file is crucial for reproducibility.
• How to parse Terraform outputs as JSON for post-deployment checks.
• When to reach for a library like python-terraform vs. raw subprocess.
• How to diagnose and fix common pitfalls like stale plans and suppressed errors.

Your next step? Take this pattern and integrate it into a CI/CD pipeline. In the next lesson, you’ll learn how to wrap this orchestration in a GitHub Actions workflow, using Python to conditionally trigger Terraform based on branch names or file changes. That’s where the true automation magic begins — no more manual cloud console clicks for every merge request.