AWS Secrets Manager with Python

You've built the app, wired up the database, and pushed it to production — but somewhere in your code, a database password, API key, or OAuth token is sitting in plain text. Hardcoded secrets are a security disaster waiting to happen: one commit to a public repo or one leaked log file and your entire infrastructure is compromised. This lesson shows you how to store secrets with AWS Secrets Manager in Python, so you can keep credentials out of your code and manage them securely, with automatic rotation and fine-grained access control. By the end, you'll be able to store, retrieve, and manage secrets programmatically — the right way.

The problem this lesson solves

Hardcoding secrets is the most common — and most dangerous — mistake in cloud development. When you embed a database password directly in your Python source code, you create several problems:

• Security risk: Anyone with access to your code repository (even a private one) can read your credentials.
• Difficulty rotating: If a secret leaks, you must manually update every place it's hardcoded.
• No audit trail: You can't tell who accessed a secret or when, making compliance hard.
• Environment inconsistency: Dev, staging, and production often need different credentials, but hardcoded values force chaos.

Storing secrets in environment variables is a step up, but they're still visible in process listings and can be accidentally exposed in logs or CI outputs. AWS Secrets Manager solves this by centralizing secret storage in a secure, encrypted service. Your Python application retrieves secrets on demand via the AWS SDK, and you control access using IAM policies. This approach keeps credentials out of your code, enables automatic rotation, and gives you a full audit trail.

Core concept / mental model

Think of AWS Secrets Manager as a secure vault in the cloud. Each secret (like a database password or API key) is stored as an encrypted object, identified by a name (e.g., prod/db/password). Your Python application acts as a key holder — it must present valid AWS credentials (via IAM) to unlock the vault and retrieve the secret.

The mental model has three parts:

1. Secret Store: A central repository where secrets live, encrypted with AWS KMS (Key Management Service).
2. IAM Authorization: Your application's AWS credentials determine which secrets it can read. IAM policies grant or deny access per secret.
3. Client SDK: The boto3 library in Python provides methods like get_secret_value and create_secret to interact with the service.

Pro tip: Secrets Manager caches secrets locally after retrieval, so you don't hit the API every time — this improves performance and reduces cost. You'll see this in the hands-on section.

How it works step by step

Storing and retrieving a secret with AWS Secrets Manager in Python follows a clear sequence:

1. Set up AWS credentials — Ensure your Python environment has valid AWS credentials (via IAM role, environment variables, or ~/.aws/credentials). This is foundational; without it, your calls will fail with an auth error.
2. Install the SDK — Install boto3, the official AWS SDK for Python, using pip.
3. Create a secret — Use the create_secret API to store a new secret. You provide a name, the secret value (a string or JSON), and optionally a KMS key.
4. Retrieve a secret — Use get_secret_value to fetch the secret. The response contains the secret string, which you can parse if it's JSON.
5. Update and rotate — When credentials change, update the secret with update_secret or configure automatic rotation via Lambda.
6. Clean up — Delete secrets you no longer need with delete_secret, and make sure you set up IAM policies to limit access.

Each step builds on the previous one. Once you've created a secret, you can retrieve it anywhere in your application — as long as your IAM role permits it.

Hands-on walkthrough

Let's get your hands dirty. This walkthrough covers the essential operations: installing the SDK, creating a secret, retrieving it, and using it in a real-world-like scenario.

Prerequisites

• Python 3.10+ installed
• AWS account with appropriate permissions (e.g., secretsmanager:CreateSecret, secretsmanager:GetSecretValue)
• AWS CLI configured (or IAM role attached) for local development

Step 1: Install boto3

Open your terminal and install the AWS SDK:

pip install boto3

Step 2: Create a secret

Create a new Python file, say create_secret.py, and add the following code to store a database password:

import boto3
import json

# Create a Secrets Manager client
client = boto3.client("secretsmanager", region_name="us-east-1")

# The secret value can be a string or a JSON structure
secret_value = json.dumps({
    "username": "admin",
    "password": "S3cureP@ssw0rd!"
})

# Create the secret
response = client.create_secret(
    Name="prod/db/password",
    SecretString=secret_value,
    Description="Database credentials for production",
    Tags=[
        {"Key": "Environment", "Value": "production"},
        {"Key": "App", "Value": "my-app"}
    ]
)

print(f"Secret ARN: {response['ARN']}")

Run the script:

python create_secret.py

Expected output (similar):

Secret ARN: arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/db/password-abc123

Step 3: Retrieve the secret in Python

Now, write a script that retrieves the secret and uses it to connect to a database. This is where you'll see the real-world power.

import boto3
import json
import psycopg2  # assume you have this installed

# Create a client
client = boto3.client("secretsmanager", region_name="us-east-1")

# Retrieve the secret
try:
    response = client.get_secret_value(SecretId="prod/db/password")
    secret_string = response["SecretString"]
    credentials = json.loads(secret_string)

    # Now use the credentials to connect to your database
    conn = psycopg2.connect(
        host="my-db.example.com",
        database="mydb",
        user=credentials["username"],
        password=credentials["password"]
    )
    print("Connected to database successfully!")
    # ... do your work ...
    conn.close()
except Exception as e:
    print(f"Error: {e}")

Pro tip: For better performance, cache the secret in memory and refresh it only when rotation occurs. AWS provides a secretsmanager-caching library (e.g., secret_cache). See an example below.

from botocore.exceptions import ClientError
from aws_secretsmanager_caching import SecretCache, SecretCacheConfig
import boto3

client = boto3.client("secretsmanager", region_name="us-east-1")
cache = SecretCache(config=SecretCacheConfig(), client=client)

# Get the secret (cached after first call)
secret = cache.get_secret_string("prod/db/password")
print("Cached secret retrieved:", secret)

Step 4: Update a secret

When the database password changes, update the secret without touching your code:

# update_secret.py
import boto3
import json

client = boto3.client("secretsmanager", region_name="us-east-1")

new_value = json.dumps({
    "username": "admin",
    "password": "NewS3cureP@ss!"
})

response = client.update_secret(
    SecretId="prod/db/password",
    SecretString=new_value
)

print(f"Secret updated: {response['ARN']}")

After running this, your application's next retrieval will get the new value (if it re-fetches).

Compare options / when to choose what

AWS Secrets Manager is not the only way to store secrets in AWS. Here’s a comparison to help you choose:

Choose Secrets Manager when you need automatic rotation, secret versioning, and strict access control. For non-sensitive config (like feature flags), Parameter Store is cheaper and sufficient.

Troubleshooting & edge cases

Even with a solid setup, things can go wrong. Here are common issues and how to fix them:

• AccessDeniedException: Your IAM role/user lacks permission. Attach a policy like secretsmanager:GetSecretValue with a resource ARN.
• ResourceNotFoundException: The secret name doesn't exist. Double-check the spelling and region.
• InvalidRequestException: You tried to create a secret that already exists, or the name is invalid. Use a unique name or update_secret instead.
• ThrottlingException: You're hitting API rate limits. Implement exponential backoff and retry, or enable caching.
• ClientError with 'secret not yet exist': You may have a race condition when creating on the fly. Always create secrets before the app starts, or handle the exception with retry.
• Region mismatch: The secret is in a different AWS region than your client. Specify the correct region_name.

Pro tip: Use a secret cache library to reduce API calls and avoid throttling under high load.

What you learned & what's next

You've now mastered the core idea behind storing secrets with AWS Secrets Manager in Python. You can explain why hardcoded secrets are dangerous, describe the mental model of a secure vault, and you've completed a hands-on exercise where you created, retrieved, and updated a secret using Python's boto3. You also know how to compare Secrets Manager with alternatives like Parameter Store, and you can troubleshoot common errors like access denied and throttling.

Next lesson: In the next step of the AWS Cloud & DevOps with Python track, you'll learn how to automate secret rotation using AWS Lambda, or perhaps dive into IAM policies to lock down access even further. This foundation is critical for building secure, production-ready Python applications.