Run Shell Commands from Python

You've just spent 20 minutes manually SSH-ing into a server, typing the same systemctl restart nginx and tail -f /var/log/nginx/error.log commands you've typed a thousand times before. It's tedious, error-prone, and — let's be honest — not a great use of a DevOps engineer's brain. The good news? Python can automate all of that. In this lesson, you'll learn how to run shell commands from Python using the subprocess module, capture their output, handle errors gracefully, and choose the right execution strategy for your automation scripts. By the end, you'll be writing Python scripts that do the heavy lifting for you.

The problem this lesson solves

DevOps and SRE work is full of repetitive shell tasks: checking service status, copying logs, restarting processes, parsing ps output, or polling kubectl get pods. Doing these by hand is slow and leaky — you forget a flag, mistype a path, or miss a critical error. But your automation scripts also need to talk to the operating system, and Python's built-in tools for that are often misunderstood.

A common pitfall is reaching for os.system() or os.popen() because they look simple. They are simple, but they give you almost no control: you can't capture output directly, you can't pass arguments safely without shell injection risks, and error handling is clunky at best. This lesson replaces those crutches with the standard-library workhorse: the subprocess module.

Why this matters right now: as you build bigger automation pipelines (whether it's a CI/CD job, a deployment script, or a server-cleanup routine), you will need to run shell commands from Python reliably, capture their output, check their exit codes, and feed that info into your logic. Doing that wrong leads to silent failures, partial deployments, and hard-to-debug logs.

Core concept / mental model

Think of Python as the conductor of an orchestra. Each shell command (ls, grep, systemctl) is a musician that plays a single note. The subprocess module is the sheet music — it tells the orchestra what to play, how loud, and when to stop. Instead of just sending a command and hoping for the best, subprocess gives you three things: the command to run, the environment it runs in, and the ability to read the result.

Key terms

• Process: an instance of a running program (like grep or ping).
• Exit code: a number returned by a process when it finishes. 0 usually means success, any non-zero means an error.
• Standard output (stdout): the text a program writes to the screen.
• Standard error (stderr): the text a program writes for error messages.
• Shell: the command interpreter (like bash or sh) that turns a string into a process.

The subprocess module in one diagram

Python script ⟶ subprocess.run() ⟶ creates child process ⟶ runs command ⟶ captures stdout/stderr/exit_code ⟶ returns CompletedProcess

The key takeaway: subprocess.run() is your main function. It runs a command, waits for it to finish, and gives you a CompletedProcess object with all the info you need.

How it works step by step

Running a shell command from Python is surprisingly straightforward once you break it down. The core pattern with subprocess.run() looks like this:

1. Pick your command representation — either a list of arguments (["ls", "-l", "/tmp"]) or a single string (only when shell=True).
2. Choose whether to use the shell — shell=False by default and safest.
3. Decide what to capture — set capture_output=True to get stdout and stderr, or stdout=subprocess.PIPE for finer control.
4. Call subprocess.run() with your arguments and options.
5. Check the exit code (usually via returncode or by raising CalledProcessError with check=True).
6. Process the captured output — decode bytes to a string and parse it as needed.

That's it. Six steps cover 90% of your needs. The next section puts those steps into practice with real, runnable code.

Hands-on walkthrough

Let's write a small Python script that runs ls -l on a directory and prints the results. Create a file called list_dir.py:

import subprocess

# Run 'ls -l' on the current directory
try:
    result = subprocess.run(["ls", "-l"], capture_output=True, text=True, check=True)
    # 'text=True' makes stdout/stderr strings instead of bytes
    print(result.stdout)
except subprocess.CalledProcessError as e:
    print("Command failed with exit code", e.returncode)
    print(e.stderr)

When you run this script (python list_dir.py), you'll see the same output as running ls -l in your terminal, but now it's a string variable you can manipulate.

Capturing output for conditional logic

Often you want to use the output rather than just print it. For example, check if a certain process is running:

import subprocess

ps = subprocess.run(["pgrep", "-f", "nginx"], capture_output=True, text=True)

if ps.returncode == 0:
    print("nginx is running with PIDs:")
    print(ps.stdout)
else:
    print("nginx is NOT running")

The exit code does the heavy lifting: pgrep returns 0 if it found a match, 1 if not.

Handling environment variables and working directory

Sometimes your command needs a different environment or a different working directory:

import subprocess, os

# Set extra environment variables
env = os.environ.copy()
env["MY_VAR"] = "hello"

# Run a command in a specific directory
result = subprocess.run(
    ["pwd"],
    cwd="/tmp",          # working directory
    env=env,             # environment variables
    capture_output=True,
    text=True
)

print("Exit code:", result.returncode)
print("Output:", result.stdout.strip())

Expected output:

Exit code: 0
Output: /tmp

Real-world example: system check script

Here's a mini DevOps script that checks disk space and service status:

import subprocess

def run_cmd(cmd):
    """Run a shell command and return (exit_code, stdout, stderr)."""
    result = subprocess.run(cmd, capture_output=True, text=True)
    return result.returncode, result.stdout, result.stderr

# Check disk usage
exit_code, out, err = run_cmd(["df", "-h", "/tmp"])
if exit_code == 0:
    print(out)
else:
    print("Disk check failed:", err)

# Check if nginx process exists
exit_code, out, err = run_cmd(["pgrep", "-x", "nginx"])
if exit_code == 0:
    print("Nginx is up")
else:
    print("Nginx is down")

Pro tip: Always use text=True when you want string output. Without it, you get bytes that often need .decode('utf-8').

Compare options / when to choose what

While subprocess.run() is the go-to, Python offers several other ways to run shell commands. Here's a comparison to help you choose:

For 95% of DevOps automation, subprocess.run() is the right choice. Use Popen only when you need to stream output or feed input incrementally.

When to use shell=True (and when not to)

• Avoid shell=True when the command is static or uses user input — it opens you to shell injection.
• Use shell=True only when you need shell features like pipes, redirection, or environment variable expansion in a quick script, and you trust the input.
• Prefer passing a list of arguments (no shell) — it's safer and faster.

Troubleshooting & edge cases

Command not found

If you try to run a command that doesn't exist, subprocess.run() raises FileNotFoundError. Use a try/except:

import subprocess

try:
    subprocess.run(["nonexistentcmd"], check=True)
except FileNotFoundError:
    print("Command not found")

Exit code is non-zero but no error raised

By default, subprocess.run() does not raise an error on a non-zero exit code. Always check result.returncode or use check=True to raise CalledProcessError.

Output is in bytes

If you forgot text=True, you'll get bytes. Convert with .decode() or better, just add text=True.

Long-running commands deadlock

If a command outputs more than a pipe buffer to stdout/stderr and you didn't capture them, your script might block. Using capture_output=True handles this, but for very long outputs, consider capturing to a file.

Shell injection with user input

Building commands as strings with shell=True and user input is a security hole. This is the classic import os; os.system("ls " + user_input) disaster. Always pass a list to avoid shell injection.

What you learned & what's next

You now know how to run shell commands from Python using subprocess.run(), capture stdout/stderr, check exit codes, and handle common pitfalls. You can apply this to automate repetitive shell tasks, check system health, and integrate shell commands into larger Python automation pipelines. This is a cornerstone skill for DevOps automation — you'll use it in almost every serious script.

Great! The next lesson in this track is Managing environment variables in Python (lesson 7), where you'll learn to move beyond hard-coded configs and make your scripts portable across environments. See you there!