Encrypt S3 with KMS

Your S3 bucket is a fortress, but the treasure inside is only as safe as the lock on the chest. By default, Amazon S3 encrypts your objects with server-side encryption (SSE-S3), but it holds the keys for you. That's convenient, but if you need to control who can decrypt the data, audit key usage, or meet compliance requirements, you need your own keys. That's where encrypting S3 with KMS at rest comes in — it gives you full control over the encryption keys protecting your most sensitive data.

The problem this lesson solves

Without customer-managed keys, your data is encrypted, but the keys are managed by AWS. This is a problem because:

• You can't audit who used the key — AWS uses SSE-S3 keys invisibly, so you can't see who decrypted what.
• You can't rotate the key on your schedule — AWS rotates SSE-S3 keys automatically, but you have no control over the cadence.
• You can't revoke access quickly — If a key is compromised or you need to cut off access, you can't disable it.
• Compliance frameworks often require customer-controlled keys — HIPAA, PCI-DSS, and other standards frequently mandate that you manage encryption keys.

When you encrypt S3 with KMS at rest, you take control. You create a customer-managed key (CMK) in AWS Key Management Service (KMS), use it to encrypt your S3 objects, and manage everything about that key — rotation, policy, disabling, and deletion — with full audit trails in CloudTrail.

Core concept / mental model

Think of KMS as a vault that stores master keys. When you encrypt an S3 object with a KMS key, you don't encrypt the object directly with the master key. Instead, KMS generates a data key that does the actual encryption. This is called envelope encryption:

1. S3 asks KMS for a data key.
2. KMS returns a plaintext data key and an encrypted version of that key.
3. S3 encrypts the object with the plaintext data key.
4. S3 keeps only the encrypted data key alongside the encrypted object, and discards the plaintext key.
5. To decrypt, S3 sends the encrypted data key to KMS, which decrypts it using the master key.

This separation gives you the best of both worlds: you can encrypt large objects efficiently (using fast symmetric data keys) while controlling access to the master key in KMS. You can allow or deny who can use the key, and you can audit every call.

Pro tip: Envelope encryption is a standard pattern in cryptography. It's not just S3 — you'll see the same idea in AWS EBS, RDS, and other services that integrate with KMS.

How it works step by step

Let's walk through the exact flow of encrypting an S3 object with KMS at rest. We'll use the AWS CLI and Python's boto3 in the hands-on section, but this is the mental sequence:

1. Create a customer-managed key (CMK) in KMS. This is your master key. Set its alias and key policy.
2. Configure S3 bucket encryption to use that CMK. You can set this at the bucket level (default encryption) so every new object is automatically encrypted with your CMK.
3. Upload an object — S3 automatically calls KMS to get a data key, encrypts the object, and stores the encrypted data key with the object.
4. Decrypt the object — when you retrieve the object, S3 calls KMS to decrypt the data key, then returns the plaintext object.
5. Audit — every KMS GenerateDataKey and Decrypt call is logged in CloudTrail, so you know exactly when and who accessed your data.

One important detail: encryption at rest happens when S3 stores the object. The object is never stored in plaintext on disk. Whether you're reading or writing, S3 handles the KMS calls transparently in the background.

Hands-on walkthrough

Let's make it real. We'll use the AWS CLI and Python with boto3 to encrypt S3 with KMS at rest.

Prerequisites

• An AWS account with appropriate IAM permissions.
• AWS CLI installed and configured (aws configure).
• Python 3.10+ with boto3 installed (pip install boto3).

Step 1: Create a KMS key

# Create a KMS key and capture its ID
export KMS_KEY_ID=$(aws kms create-key --description "My S3 encryption key" --region us-east-1 --query 'KeyMetadata.KeyId' --output text)

# Set an alias for easier reference
export KMS_KEY_ARN="arn:aws:kms:us-east-1:$(aws sts get-caller-identity --query Account --output text):key/$KMS_KEY_ID"
aws kms create-alias --alias-name alias/my-s3-key --target-key-id $KMS_KEY_ID

echo "Key created: $KMS_KEY_ARN"

Step 2: Create a bucket and enable default encryption with KMS

export BUCKET_NAME="my-kms-encrypted-bucket-$(date +%s)"
aws s3api create-bucket --bucket $BUCKET_NAME --region us-east-1

aws s3api put-bucket-encryption \
  --bucket $BUCKET_NAME \
  --server-side-encryption-configuration "{
    \"Rules\": [
      {
        \"ApplyServerSideEncryptionByDefault\": {
          \"SSEAlgorithm\": \"aws:kms\",
          \"KMSMasterKeyID\": \"$KMS_KEY_ID\"
        },
        \"BucketKeyEnabled\": true
      }
    ]
  }"

echo "Bucket $BUCKET_NAME created and encrypted with KMS."

Step 3: Upload and decrypt using Python with boto3

Now we'll use Python to upload an object and verify that it's encrypted with our CMK. Note that we don't have to pass any special parameters because the bucket-level default encryption applies automatically.

import boto3
import os

s3 = boto3.client("s3", region_name="us-east-1")

bucket = "my-kms-encrypted-bucket-1234567890"  # replace with your bucket
object_key = "confidential/report.pdf"

# Write a temporary file to upload
with open("/tmp/report.pdf", "wb") as f:
    f.write(b"Top secret: quarterly earnings for Q3.")

# Upload the object (default encryption with KMS applies automatically)
s3.upload_file(
    Filename="/tmp/report.pdf",
    Bucket=bucket,
    Key=object_key,
    ExtraArgs={"ServerSideEncryption": "aws:kms"}  # optional, but explicit
)
print(f"Uploaded and encrypted at rest with KMS.")

# Check that the object is encrypted with KMS
head = s3.head_object(Bucket=bucket, Key=object_key)
print(f"ServerSideEncryption: {head.get('ServerSideEncryption')}")
print(f"KMSKeyId: {head.get('SSEKMSKeyId')}")

# Download and read the decrypted content (S3 transparently decrypts)
s3.download_file(Bucket=bucket, Key=object_key, Filename="/tmp/decrypted.pdf")
with open("/tmp/decrypted.pdf", "r") as f:
    print(f"Decrypted content: {f.read()}")

Expected output (approximate):

Uploaded and encrypted at rest with KMS.
ServerSideEncryption: aws:kms
KMSKeyId: arn:aws:kms:us-east-1:123456789012:key/abc12345-...
Decrypted content: Top secret: quarterly earnings for Q3.

Pro tip: Enable S3 Bucket Keys (we did above with BucketKeyEnabled: true) to reduce KMS API call costs — S3 will use a short-lived bucket key instead of calling KMS for every object. This is a best practice for cost optimization.

Compare options / when to choose what

You have several ways to encrypt S3 objects at rest. Let's compare the most common choices:

When should you choose encrypt S3 with KMS at rest?

• Choose SSE-KMS with a customer-managed CMK when you need key control, rotation, and audit trails — typical for financial data, health records, or any regulated workload.
• Choose SSE-KMS with an AWS-managed key (the default when you enable default encryption with KMS) when you want KMS features but don't want to manage key lifecycle.
• Choose SSE-S3 when your data isn't sensitive and you just want a low-cost, zero-config encryption.
• Choose client-side encryption (like the AWS Encryption SDK) when you need to decrypt data outside of AWS or want to ensure nobody at AWS can ever see your unencrypted data.

Troubleshooting & edge cases

1. AccessDenied when uploading or decrypting

Cause: The IAM role/user lacks permission to use the KMS key for kms:GenerateDataKey or kms:Decrypt.

Fix: Attach a policy like this to the IAM entity:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/your-key-id"
    }
  ]
}

Also check the key policy — if the key policy denies access, no IAM policy can override it.

2. Objects upload but fail to download with KMS.NotFoundException

Cause: The CMK was deleted or is currently disabled.

Fix: Re-enable the key or restore a deleted key (if within the waiting period). If the key is permanently deleted, you can't decrypt the objects — this is a permanent loss.

Pro tip: Enable KMS key deletion waiting period (7–30 days) and never disable a key while data is encrypted with it. Always test rotation in a non-production environment first.

3. InvalidArgument when setting bucket encryption

Cause: The IAM user/role lacks permission to call kms:CreateGrant on the key.

Fix: Ensure the caller has kms:CreateGrant plus kms:DescribeKey on the key. S3 needs a grant to use the key on your behalf.

4. KMS.ThrottlingException when uploading many objects

Cause: KMS has a rate limit of 5,500 requests per second (default). If you're uploading thousands of objects, you may hit the limit.

Fix: Enable S3 Bucket Keys (reduces KMS calls massively) or increase your KMS request quota via Service Quotas.

What you learned & what's next

You now know how to encrypt S3 with KMS at rest: you create a customer-managed KMS key, configure S3 bucket default encryption with that key, and upload objects that are automatically encrypted and decrypted transparently. You've seen how envelope encryption works, compared it with other options, and troubleshooted common pitfalls like permissions and key lifecycle issues.

You also learned how to audit key usage via CloudTrail, and why managing your own keys is critical for compliance and cloud security. This is a cornerstone skill in cloud security essentials — you no longer have to trust AWS to hold your keys.

What's next?

In the next lesson, you'll learn how to rotate KMS keys and manage key lifecycle, building on your ability to encrypt data at rest. You'll also explore how to protect S3 data in transit with S3 bucket policies and Access Points. Stay tuned!

Now go ahead and encrypt your own bucket — practice makes perfect.