Mock S3 List Objects Paginator in Python

This code implements a mock S3 paginator that yields pages of object keys, mimicking the behavior of boto3's list_objects_v2 paginator for local testing.

Medium Python 3.9+ Aug 9, 2026 Cloud + Python 13 views 0 copies

s3 mock paginator aws testing boto3

Python code

43 lines
Python 3.9+
Run it 
import json
from datetime import datetime, timezone


class MockS3Paginator:
    """A mock S3 list_objects_v2 paginator returning pages of keys."""

    def __init__(self, bucket, all_keys, page_size=1000):
        self.bucket = bucket
        self.all_keys = all_keys
        self.page_size = page_size

    def paginate(self, **kwargs):
        start = 0
        while start < len(self.all_keys):
            end = min(start + self.page_size, len(self.all_keys))
            page_keys = self.all_keys[start:end]
            yield {
                "IsTruncated": end < len(self.all_keys),
                "Contents": [
                    {
                        "Key": key,
                        "Size": len(key.encode()),
                        "LastModified": datetime.now(timezone.utc).isoformat(),
                    }
                    for key in page_keys
                ],
                "MaxKeys": self.page_size,
                "NextContinuationToken": str(end) if end < len(self.all_keys) else None,
            }
            start = end


if __name__ == "__main__":
    keys = [f"data/file_{i:03d}.csv" for i in range(5)]
    paginator = MockS3Paginator("my-bucket", keys, page_size=2)

    all_found = []
    for page in paginator.paginate(Prefix="data/"):
        all_found.extend(obj["Key"] for obj in page["Contents"])
        print(json.dumps({"truncated": page["IsTruncated"], "count_on_page": len(page["Contents"])}))

    print("ALL_KEYS:", json.dumps(all_found))

Output

stdout 
{"truncated": true, "count_on_page": 2}
{"truncated": true, "count_on_page": 2}
{"truncated": false, "count_on_page": 1}
ALL_KEYS: ["data/file_000.csv", "data/file_001.csv", "data/file_002.csv", "data/file_003.csv", "data/file_004.csv"]

How it works

This mock paginator follows the generator pattern: each call to paginate returns a generator that yields page dictionaries. The paginate method accepts a Prefix keyword argument (ignored here for simplicity) to mimic boto3's interface. Each page includes IsTruncated, Contents, MaxKeys, and NextContinuationToken keys, matching the real S3 response structure. The code slices the full list of keys by page_size, ensuring each page contains at most page_size items, and the IsTruncated flag reflects whether more pages remain. This allows you to write and test code that consumes paginated S3 responses without hitting AWS.

Common mistakes

  • Forgetting that `paginate` returns a generator, not a list; you must iterate over it.
  • Assuming the `Prefix` argument filters results; this mock ignores it and returns all keys.
  • Omitting `NextContinuationToken` on the last page; real S3 returns `None` here.
  • Using `len(key)` instead of `len(key.encode())` for byte size, which miscounts multi-byte characters.

Variations

  1. Use `yield from` to flatten pages into a single key stream.
  2. Add `Prefix` filtering inside `paginate` to match real S3 behavior.
  3. Return an empty list for `Contents` when there are no keys.

Real-world use cases

  • Unit-testing an S3 client wrapper that calls `list_objects_v2` and accumulates keys across pages.
  • Simulating large S3 bucket listings in local development environments without network access.
  • Prototyping data pipeline logic that processes objects in batches before deploying to AWS.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Cloud + Python

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes