Offset vs Keyset Pagination in Python
Demonstrate offset-based pagination and keyset (cursor) pagination with a simple in-memory dataset, showing how each returns pages of records.
Python code
36 lines"""Demonstrate pagination using offset vs keyset (cursor) approach."""
ITEMS = [
{"id": 1, "name": "Alice"},
{"id": 2, "name": "Bob"},
{"id": 3, "name": "Carol"},
{"id": 4, "name": "David"},
{"id": 5, "name": "Eve"},
]
def offset_paginate(items, page, page_size):
"""Return a page using offset-based pagination."""
start = (page - 1) * page_size
return items[start:start + page_size]
def keyset_paginate(items, last_id, page_size):
"""Return a page using keyset (cursor) pagination, assuming sorted by id."""
if last_id is None:
return items[:page_size]
start = next(i for i, item in enumerate(items) if item["id"] == last_id) + 1
return items[start:start + page_size]
if __name__ == "__main__":
print("Offset pagination:")
for page in range(1, 4):
result = offset_paginate(ITEMS, page, page_size=2)
print(f" page {page}: {[item['name'] for item in result]}")
print("Keyset pagination:")
cursor = None
for _ in range(3):
result = keyset_paginate(ITEMS, cursor, page_size=2)
if not result:
break
print(f" after {cursor}: {[item['name'] for item in result]}")
cursor = result[-1]["id"]
Output
Offset pagination:
page 1: ['Alice', 'Bob']
page 2: ['Carol', 'David']
page 3: ['Eve']
Keyset pagination:
after None: ['Alice', 'Bob']
after 2: ['Carol', 'David']
after 4: ['Eve']
How it works
Offset pagination calculates a slice by multiplying (page-1) * page_size to find the start index, then uses Python slicing to return that page. Keyset pagination tracks the last item's unique ID (cursor) and finds the next page by locating that ID and slicing from the following element. Offset is simple but can slow down on large tables because the database must skip many rows. Keyset is more efficient for large datasets because it uses indexed columns to jump directly to the next batch, avoiding costly OFFSET scans.
Common mistakes
- Forgetting that offset pages can show duplicate or missing rows when new records are inserted between requests
- Using keyset pagination on a column that is not unique or not sorted with a stable order
- Assuming `items` is sorted by the cursor column; keyset requires a deterministic ordering to work correctly
Variations
- Use a database index (e.g., on `id`) and a WHERE clause like `WHERE id > last_id ORDER BY id LIMIT page_size` for keyset in SQL
- Return the next cursor in the API response so clients can pass it back for the next page request
Real-world use cases
- Implementing a REST API endpoint that lists user records — offset is fine for small tables, keyset scales better for millions of rows.
- Building an infinite scroll feed that needs stable ordering without page drift as new items are added.
- Writing a nightly sync script that pages through a large database table without memory blow-ups on each request.
Sponsored
More from Database scaling & optimization
- Approximate Count with HyperLogLog in Python medium
- B-Tree Insert and In-Order Traversal in Python hard
- Broadcast a Small Reference Table in Python easy
- Build a Full Text Search Index in Python medium
- Build a Partial Index Mock in Python for Database Filtering easy
- Composite index leftmost prefix in Python medium
Keep learning
Related tutorials and quizzes for this topic.