Sync Files to S3 with AWS CLI

You've mastered creating buckets, uploading individual files, and setting permissions — but now you're facing the real-world mess: hundreds of local files that need to reach S3 without babysitting each upload. Manually running aws s3 cp for every file is tedious, error-prone, and a surefire way to overwrite newer files with stale ones. The aws s3 sync command eliminates that pain by comparing source and destination, transferring only what changed, and deleting what's no longer needed — all in one deterministic command.

The problem this lesson solves

If you've ever used aws s3 cp to upload a folder, you know the friction: it uploads every file, even ones that haven't changed. That wastes bandwidth, slows down your deploy, and makes incremental backups awkward. Worse, if you delete a file locally, cp won't remove it from S3 — so your bucket slowly becomes a graveyard of stale assets.

The AWS CLI sync command is the fix. It compares the local directory with the S3 prefix, then pushes only new or modified files. With the right flags, it can also delete remote files that no longer exist locally, and even prune empty folders. This turns S3 uploads into a fast, repeatable, and scriptable operation — exactly what you need for deployment pipelines, backup jobs, and content publishing.

Core concept / mental model

Think of aws s3 sync like a two-way mirror (but with a one-way default). It examines the source (typically a local folder) and the destination (an S3 bucket prefix) and reconciles them based on:

• File presence: If a file exists locally but not in S3, it's uploaded.
• Size and modification time: If a file exists in both but differs (size or LastModified), it's uploaded to make the destination match.
• Deletion (only with --delete): If a file exists in S3 but not locally, it's removed.

The command is stateless — it doesn't keep a manifest or database. It performs a listing of both sides on every run and computes a diff. That's why it's safe to run repeatedly: it's idempotent (running it twice with no changes does nothing).

Pro tip: Think of it as rsync for S3, but with S3's own rules for modification time (S3 stores LastModified, not Unix mtime). Keep that in mind when syncing from filesystems with odd timestamps.

How it works step by step

Here's what happens when you run aws s3 sync:

1. List the source — the CLI recursively lists all objects in your local folder.
2. List the destination — it lists all objects under the S3 prefix (optionally with --include/--exclude filters).
3. Compute the diff — compares relative paths, sizes, and timestamps.
4. Upload — files that are new or changed get uploaded (multipart for large files).
5. Delete (optional) — with --delete, files that exist in S3 but not locally are removed.
6. Report — prints a summary of transfers and deletions.

The comparison logic for a file to upload:

def needs_upload(local_file, s3_object):
    if s3_object is None:
        return True  # doesn't exist in S3
    if local_file.size != s3_object.size:
        return True  # size changed
    # AWS CLI uses a threshold: if size differs OR local mtime is newer than S3 LastModified
    if local_file.mtime >= s3_object.last_modified:
        return True  # local is newer
    return False

Hands-on walkthrough

Prerequisites

Make sure you have the AWS CLI installed and configured (if not, review earlier lessons):

aws --version
aws configure  # sets access key, secret, region

Basic sync: upload local folder to S3

Create a local folder with a few files, then sync it to a bucket:

mkdir -p ~/my-site/assets
 echo 'body { color: red; }' > ~/my-site/assets/style.css
 echo 'console.log("hi")' > ~/my-site/assets/app.js

aws s3 sync ~/my-site/assets s3://my-bucket/assets/

Expected output (first run):

upload: assets/style.css to s3://my-bucket/assets/style.css
upload: assets/app.js to s3://my-bucket/assets/app.js

Now modify style.css and run again — only that file uploads:

echo 'body { color: blue; }' > ~/my-site/assets/style.css
aws s3 sync ~/my-site/assets s3://my-bucket/assets/

Output: only style.css is uploaded.

Sync with deletion (mirror your local folder)

If you delete a local file and want S3 to match, use --delete:

rm ~/my-site/assets/app.js
aws s3 sync ~/my-site/assets s3://my-bucket/assets/ --delete

Output includes a delete: line.

Sync with filters (exclude temp files)

Use --exclude and --include to skip logs or caches:

aws s3 sync . s3://my-bucket/site/ --exclude "*.tmp" --exclude "node_modules/*"

You can combine multiple filters; order matters — the first matching pattern wins.

Sync from S3 to local (download)

The command is symmetric — swap source and destination:

aws s3 sync s3://my-bucket/backups/ ./backups/

Full example with flags

aws s3 sync ./build/ s3://my-app-frontend/ --delete --exclude "*.map" --exclude "*.DS_Store" --size-only

• --size-only compares only file size, ignoring timestamps (useful if mtimes are off).
• --exclude prevents uploading source maps and macOS metadata.
• --delete prunes stale files.

Compare options / when to choose what

For most workflows, sync is the go-to because it's efficient and safe. Use cp when you explicitly want to overwrite unconditionally (e.g., forcing a re-upload of corrupted files). Use s3api when you need to set metadata or encryption at upload time.

Troubleshooting & edge cases

• Accidental deletions — Running sync --delete without thinking can wipe files you meant to keep. Always do a dry run first: bash aws s3 sync ./build s3://bucket --delete --dryrun
• Newer local file doesn't upload — If your local mtime is older than S3 LastModified, the CLI skips it. Use --size-only to force uploads when content changed but timestamps are misleading.
• Permission errors — Upload failed: AccessDenied. Check your IAM policy: you need s3:PutObject, s3:ListBucket, and optionally s3:DeleteObject for --delete.
• S3 LastModified is updated on copy or encryption changes — Running sync can re-upload files even if content is identical if S3's timestamp changed. Use --size-only to avoid needless transfers.
• Many small files — Sync can be slow with thousands of tiny files. Use aws s3 sync --multipart-upload flags or consider compressing into larger files and using --only-show-errors for cleaner output.

What you learned & what's next

You now know how to sync files to S3 with the AWS CLI — comparing local and remote state, transferring only diffs, and optionally deleting stale objects. You saw how filters control what gets transferred, and how --dryrun prevents costly mistakes. This is the foundation for automated deploys and backups.

Next lesson in the AWS Tutorial track covers S3 event notifications and Lambda triggers — where synced files become the spark for serverless processing. With sync, you can reliably push data, and events let you react to it.