Store Secrets in AWS Secrets Manager

You’ve built a Python app, tested it locally, and now it’s time to deploy to AWS. But where do you put your database passwords, API keys, and OAuth tokens? Hardcoding them is a security disaster, and storing them in plain text files is only slightly better. In this lesson, you’ll learn how to store Python app secrets in AWS Secrets Manager, the AWS service designed to keep your sensitive data encrypted, rotated, and accessible to your code only when it needs it. By the end, you’ll be able to securely fetch secrets in Python, compare Secrets Manager with other options, and avoid common pitfalls.

The problem this lesson solves

Every application has secrets: database credentials, API keys, encryption keys, and more. If you hardcode them into your source code, they end up in your version control history, visible to anyone with repo access. If you put them in environment variables, they can leak through logs, CI pipelines, or developer laptops. And if you store them in plain text files like config.ini, a single misconfigured server can expose them to the world.

Pro tip: A secret leaked is a breach waiting to happen. Even if the secret is for a staging environment, treat it with the same care as production credentials.

AWS Secrets Manager solves this by centralizing secret storage, encrypting it at rest with AWS KMS, and providing an API to retrieve secrets on demand. You can also enable automatic rotation, so your database passwords change regularly without manual effort. This lesson focuses on how to use Secrets Manager from Python, but the concepts apply to any application running on AWS.

Core concept / mental model

Think of Secrets Manager as a vault with a keyhole. The vault (Secrets Manager) holds your secrets in encrypted form. The keyhole is the AWS SDK, which uses your IAM permissions to open the vault and hand you the secret — but only if you’re allowed. No secret is stored in your code, and no secret is readable by anyone without the right permissions.

Here’s a simple diagram in words:

Python app --> AWS SDK (boto3) --> Secrets Manager --> Returns secret as JSON

The SDK authenticates on your behalf using the credentials available in the environment (e.g., IAM role on EC2, access keys locally). Once authenticated, it calls get_secret_value and retrieves the secret as a string (usually JSON). Your app then parses that JSON to extract the fields it needs.

Key terms to remember:

• Secret: A key-value pair or arbitrary string stored in Secrets Manager.
• Secret ARN: The Amazon Resource Name that uniquely identifies the secret.
• Rotation: The process of automatically updating the secret value periodically.
• KMS key: The encryption key used to encrypt the secret at rest.

How it works step by step

Let’s walk through the lifecycle of using a secret with Secrets Manager:

1. Create the secret — Use the AWS Console, CLI, or SDK to store your secret. For example, you might store a JSON object with username and password for a database.
2. Grant IAM permissions — Your Python app needs an IAM role or user with permission to read the secret. The least-privilege policy would be secretsmanager:GetSecretValue on the specific secret ARN.
3. Retrieve the secret in Python — Use boto3 to call get_secret_value(). The SDK handles authentication, then returns the secret string.
4. Parse and use — Convert the secret string to a Python dictionary and use the values in your database connection or API call.
5. (Optional) Rotate the secret — Set up a Lambda function to update the secret value on a schedule. Your app just keeps fetching the latest value.

This flow ensures that secrets are never in code, and they can be updated without redeploying your application. The key insight is that you only fetch the secret when you need it, and you can cache it for performance if needed.

Hands-on walkthrough

Let’s get practical. You’ll need boto3 installed, and AWS credentials configured (either via AWS CLI or environment variables).

Step 1: Install boto3

pip install boto3

Step 2: Create a secret in AWS Secrets Manager

You can use the AWS CLI or the console. Here’s the CLI command to create a secret with a JSON value:

aws secretsmanager create-secret \
  --name MyDatabaseSecret \
  --secret-string '{"username":"admin","password":"SuperSecret!123"}'

Note the output includes the ARN — you’ll use that later.

Step 3: Write Python code to retrieve the secret

Create a Python script that fetches the secret and uses it.

import boto3
import json
from botocore.exceptions import ClientError

def get_secret(secret_name, region_name="us-east-1"):
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )
    try:
        response = client.get_secret_value(SecretId=secret_name)
        # The secret is returned as a string (JSON usually)
        secret_string = response.get('SecretString')
        if secret_string:
            return json.loads(secret_string)
        else:
            # If the secret is binary, handle it differently
            return response['SecretBinary']
    except ClientError as e:
        error_code = e.response['Error']['Code']
        if error_code == 'ResourceNotFoundException':
            print("Secret not found")
        elif error_code == 'AccessDeniedException':
            print("Permission denied")
        else:
            raise

# Use it
db_creds = get_secret("MyDatabaseSecret")
print(f"Connecting to DB as {db_creds['username']}")
# This is where you would use the credentials, e.g.,
# engine = create_engine(f"postgresql://{db_creds['username']}:{db_creds['password']}@host/db")

Expected output:

Connecting to DB as admin

Step 4: Handle missing secrets gracefully

In production, your code should fail with a clear message if the secret is missing, but not print the secret itself.

try:
    db_creds = get_secret("MyDatabaseSecret")
    print("Secret retrieved successfully")
except Exception as e:
    print("ERROR: Could not retrieve secret")
    # Log the error for debugging, but not the secret
    raise SystemExit(1)

Compare options / when to choose what

You have several ways to store secrets on AWS. Here’s a quick comparison to help you choose.

For most production applications, Secrets Manager is the recommended choice if you need rotation or have secrets that change. If you have a tight budget and need only static secrets, Parameter Store’s SecureString is a viable alternative. For everything else, use Secrets Manager.

Pro tip: Start with Secrets Manager even if you think you don’t need rotation. The cost is minimal for a handful of secrets, and you avoid migration later.

Troubleshooting & edge cases

Here are common issues you might face and how to solve them.

• AccessDeniedException: Your IAM role/user lacks secretsmanager:GetSecretValue permission. Attach a policy like arn:aws:iam::aws:policy/SecretsManagerReadWrite (or a custom one) to the role.
• ResourceNotFoundException: You typed the secret name wrong. Use the full ARN if you have multiple regions.
• SecretString is None: The secret might be binary. Use SecretBinary instead, but most use cases are strings.
• Caching secrets for performance: Calling get_secret_value every time adds latency. Cache the secret in memory for a few minutes, but be aware of rotation — if you cache too long, you might use a stale secret.
• Region mismatch: Secrets are regional. Ensure your Python client uses the same region as the secret.

For example, here’s a simple cache implementation:

import time
import boto3
import json

_cache = {}

def get_secret_cached(secret_name, region_name):
    now = time.time()
    if secret_name in _cache and (now - _cache[secret_name]['timestamp'] < 300):
        return _cache[secret_name]['data']
    # Fetch from AWS
    client = boto3.client('secretsmanager', region_name=region_name)
    response = client.get_secret_value(SecretId=secret_name)
    secret_data = json.loads(response['SecretString'])
    _cache[secret_name] = {'data': secret_data, 'timestamp': now}
    return secret_data

What you learned & what's next

You now know how to store Python app secrets in AWS Secrets Manager: you can create a secret, retrieve it securely in Python, and handle errors. You also learned when to prefer Secrets Manager over Parameter Store or environment variables. Finally, you saw how to cache secrets for performance while staying aware of rotation.

Now that secrets are secure, the next step in your AWS Cloud & DevOps with Python journey is learning how to deploy your Python app to AWS using services like EC2 or ECS, where you can attach IAM roles that grant permission to fetch these secrets automatically. That’s where the real power of secure cloud-native development comes together.