S3 Lifecycle Policies & Versioning

You deployed an S3 bucket, stored years of logs, and now your monthly bill is climbing while your storage is cluttered with obsolete versions of every object you ever touched. Without lifecycle policies and versioning, your S3 bucket is a silent budget killer — every overwrite and delete leaves behind data you're paying for forever. This lesson shows you how to take control with Python, using lifecycle rules to automate the entire journey from frequent access to archival to deletion, and versioning to protect against accidental loss — all without lifting a finger after the initial setup.

The problem this lesson solves

Raw S3 buckets are static: once an object is uploaded, it stays there unless you manually clean it up. In production, that means:

• Uncontrolled storage growth — log files, backups, and temporary artifacts pile up. Each GB costs money per month, and the cost compounds as data accumulates.
• Accidental overwrites and deletes — a bad script or a typo can permanently destroy data. Without versioning, DeleteObject is irreversible; PutObject overwrites the original silently.
• Manual cleanup is error-prone — scripting a nightly delete job risks deleting the wrong keys. You need a declarative, automated approach that AWS itself enforces.
• Costs are opaque — you rarely notice the creeping costs of old data until the invoice arrives. Standard storage is up to 10× more expensive than Glacier Deep Archive, yet most buckets never use it.

Imagine a log bucket that receives 10 GB of new logs daily. Without lifecycle rules, after one year that's over 3.6 TB of data sitting in S3 Standard. With a lifecycle policy moving data older than 30 days to Glacier and deleting after a year, you could cut costs by 60–80% while keeping data accessible for audits.

Core concept / mental model

Think of your bucket as a three-layer pyramid:

1. Bottom layer — data accumulated over time: logs, backups, old versions.
2. Middle layer — your lifecycle policy: the automated schedule that decides when each object transitions to cheaper storage or gets deleted.
3. Top layer — versioning: the safety net that keeps every version of every object, so a policy can never accidentally destroy your only copy.

Definitions you need to master:

• S3 lifecycle configuration: An XML (or JSON via the API) document attached to a bucket with one or more rules. Each rule has a filter (which objects it applies to), a status (Enabled or Disabled), and a set of actions (transition or expiration).
• Transition action: Moves objects to a different storage class after a specified number of days from creation (or from becoming noncurrent).
• Expiration action: Permanently deletes objects after a specified number of days.
• Abort incomplete multipart upload: Clean up orphaned upload parts after a number of days — a hidden cost culprit.
• Versioning: A bucket-level setting that, when enabled, retains every version of an object. You can list, delete, or restore any version. When versioning is enabled, DeleteObject adds a delete marker instead of destroying data.

Pro tip: Lifecycle policies are the automation, versioning is the insurance. You should usually enable both — versioning makes lifecycle rules safe, and lifecycle rules make versioning affordable (by expiring old noncurrent versions).

How it works step by step

Step 1: Enable versioning on your bucket

Versioning is a bucket-level property. You can enable it with one API call — but you cannot turn it off (you can only suspend it). Once enabled, every object version is kept forever unless you explicitly delete it or a lifecycle rule expires it.

Step 2: Define your lifecycle rule

A lifecycle rule consists of:

• Transitions: e.g., move to STANDARD_IA after 30 days, GLACIER after 90 days, DEEP_ARCHIVE after 180 days.
• Expiration: e.g., delete objects after 365 days.
• Noncurrent version actions: e.g., expire noncurrent versions after 30 days so old versions don't pile up.
• Filter: scope by prefix (e.g., logs/) or tag (e.g., department=archive).
• Status: Enabled means the rule is active. You can disable a rule instead of deleting it — useful for testing.

Step 3: Apply the policy via put_bucket_lifecycle_configuration

The rule must be serialized to XML or JSON depending on the SDK version. Boto3 expects a Python dict that it converts to XML internally. The bucket must be in a region where lifecycle policies are supported (all regions) and you must have the s3:PutLifecycleConfiguration permission.

Step 4: Monitor and verify

You can read the current configuration with get_bucket_lifecycle_configuration, and list objects with versions to see the effect. AWS applies lifecycle transitions asynchronously — typically within 24–48 hours, but expiration may take up to a few days.

Cause → effect: When a rule with Expiration: {Days: 365} is enabled, S3 schedules the deletion of any object that is older than 365 days and not newer. When versioning is on, this applies to both current and noncurrent versions if you specify the right fields.

Hands-on walkthrough

Let's get our hands dirty with boto3. First, install boto3 if you haven't already:

pip install boto3

1. Enable versioning

import boto3

s3 = boto3.client('s3')

bucket_name = 'my-devops-bucket'

# Enable versioning — this is irreversible (you can only suspend later)
s3.put_bucket_versioning(
    Bucket=bucket_name,
    VersioningConfiguration={'Status': 'Enabled'}
)

print(f"Versioning enabled on {bucket_name}")

Expected output:

Versioning enabled on my-devops-bucket

2. Add a lifecycle policy with transition and expiration

import boto3

s3 = boto3.client('s3')

bucket_name = 'my-devops-bucket'

lifecycle_config = {
    'Rules': [
        {
            'ID': 'logs-lifecycle',
            'Status': 'Enabled',
            'Filter': {'Prefix': 'logs/'},
            'Transitions': [
                {
                    'Days': 30,
                    'StorageClass': 'STANDARD_IA'
                },
                {
                    'Days': 90,
                    'StorageClass': 'GLACIER'
                }
            ],
            'Expiration': {'Days': 365},
            'NoncurrentVersionExpiration': {'NoncurrentDays': 30}
        },
        {
            'ID': 'backup-expiration',
            'Status': 'Enabled',
            'Filter': {'Prefix': 'backups/'},
            'Expiration': {'Days': 180}
        }
    ]
}

s3.put_bucket_lifecycle_configuration(
    Bucket=bucket_name,
    LifecycleConfiguration=lifecycle_config
)

print("Lifecycle policy applied successfully.")

Expected output:

Lifecycle policy applied successfully.

Pro tip: Always test in a development bucket first. Once you set Days: 0 to expire immediately, objects will start being deleted as soon as the rule is applied. Keep a backup of critical data before enabling aggressive policies.

3. Verify the policy

import boto3

s3 = boto3.client('s3')

response = s3.get_bucket_lifecycle_configuration(
    Bucket='my-devops-bucket'
)

print("Current lifecycle rules:")
for rule in response['Rules']:
    print(f"  ID: {rule['ID']} | Status: {rule['Status']} | Prefix: {rule.get('Filter', {}).get('Prefix', 'all')}")
    for transition in rule.get('Transitions', []):
        print(f"    Transition to {transition['StorageClass']} after {transition['Days']} days")
    if 'Expiration' in rule:
        print(f"    Expiration after {rule['Expiration']['Days']} days")

Expected output:

Current lifecycle rules:
  ID: logs-lifecycle | Status: Enabled | Prefix: logs/
    Transition to STANDARD_IA after 30 days
    Transition to GLACIER after 90 days
    Expiration after 365 days
  ID: backup-expiration | Status: Enabled | Prefix: backups/
    Expiration after 180 days

Compare options / when to choose what

Key decision factors:

• Do you need to recover from accidental deletion? → Enable versioning.
• Do you need to keep data for a fixed period? → Use lifecycle with expiration — but ensure your compliance needs allow deletion.
• Do you want to save costs on old data? → Use transitions to cheaper storage classes. GLACIER and DEEP_ARCHIVE have lower storage costs but retrieval fees — only use for cold data.
• Multipart uploads get abandoned? → Add a rule with AbortIncompleteMultipartUpload: { DaysAfterInitiation: 7 } — this is a classic cost leak.

Alternatives to lifecycle policies:

• Glacier Vault Lock — for compliance, it locks policies to prevent deletion (but you don't manage per-object transitions via Python as easily).
• Lambda event handlers — trigger a function on ObjectCreated to move objects — but that's event-driven and has more moving parts. Lifecycle is native, asynchronous, and cost-effective.
• S3 Intelligent-Tiering — automatically moves objects between tiers based on access patterns, but it's not a substitute for lifecycle rules that have specific retention requirements. It's simpler but less predictable for archival.

Troubleshooting & edge cases

1. PutBucketLifecycleConfiguration not allowed

Error: ClientError: An error occurred (AccessDenied) when calling the PutBucketLifecycleConfiguration operation: Access Denied

Cause: Your IAM user/role lacks s3:PutLifecycleConfiguration permission.

Fix: Attach an IAM policy like:

{
  "Effect": "Allow",
  "Action": ["s3:PutLifecycleConfiguration", "s3:GetLifecycleConfiguration"],
  "Resource": "arn:aws:s3:::my-devops-bucket"
}

2. Rule doesn't seem to apply

Symptom: Objects aren't transitioning or expiring after 48 hours.

Causes: - The rule is Disabled — double-check Status. - The objects don't match the prefix filter — ensure your Filter uses the exact prefix (e.g., logs/ vs logs). - You're checking standard storage class transitions — STANDARD_IA minimum object size is 128 KB. Objects smaller than that won't transition but will still expire. - Regional availability — DEEP_ARCHIVE is not available in every region.

3. Versioning can't be disabled but can be suspended

Once versioning is enabled, calling put_bucket_versioning with Suspended will stop creating new versions, but existing versions remain. You can't restore objects that were deleted with a delete marker after you overwrite it — so always test with non‑critical data.

4. Expiration of versions vs. delete markers

When you expire a noncurrent version, S3 deletes that version permanently. But if you only delete the current version, a delete marker is created and the old version becomes noncurrent — which you then need your NoncurrentVersionExpiration rule to clean up.

5. Transitioning to GLACIER with restore times

Expect retrieval to take minutes to hours. If your application needs immediate access, don't transition to GLACIER. Use STANDARD_IA or INTELLIGENT_TIERING instead.

What you learned & what's next

You now know how to:

• Explain the core idea behind lifecycle policies and versioning: declarative automation of storage class transitions and deletion, plus version retention for safety.
• Apply them in a practical exercise with boto3 — enabling versioning, adding lifecycle rules, and verifying the configuration.
• Troubleshoot common issues like permissions, filters, and object size limits.

Versioning and lifecycle policies are the foundation of a cost-efficient, resilient S3 strategy. With these in place, you can confidently move to S3 event notifications and Lambda triggers — where you'll wire your bucket to react to object events and automate downstream processing. That's the next lesson in this track.

Next up: S3 Event Notifications with Lambda — we'll build a serverless pipeline that processes files the moment they land in your bucket.