Batch Rows in Chunks with a Generator in Python

Group a list of row dicts into fixed-size chunks using a generator that yields one slice per call.

Easy Python 3.9+ Aug 9, 2026 Comprehensions & generators 14 views 0 copies

generators chunking database iteration batching

Python code

21 lines

Python 3.9+

from typing import Iterator, List


def batch_rows(rows: List[dict], batch_size: int) -> Iterator[List[dict]]:
    for i in range(0, len(rows), batch_size):
        yield rows[i:i + batch_size]


if __name__ == "__main__":
    sample_rows = [
        {"id": 1, "name": "Alice"},
        {"id": 2, "name": "Bob"},
        {"id": 3, "name": "Carol"},
        {"id": 4, "name": "Dave"},
        {"id": 5, "name": "Eve"},
        {"id": 6, "name": "Frank"},
        {"id": 7, "name": "Grace"},
    ]

    for index, chunk in enumerate(batch_rows(sample_rows, batch_size=3), start=1):
        print(f"Chunk {index}: {chunk}")

Output

stdout

Chunk 1: [{'id': 1, 'name': 'Alice'}, {'id': 2, 'name': 'Bob'}, {'id': 3, 'name': 'Carol'}]
Chunk 2: [{'id': 4, 'name': 'Dave'}, {'id': 5, 'name': 'Eve'}, {'id': 6, 'name': 'Frank'}]
Chunk 3: [{'id': 7, 'name': 'Grace'}]

How it works

The loop uses range(0, len(rows), batch_size) to step by the batch size, so each index marks the start of a chunk. Slicing with rows[i:i + batch_size] grabs exactly up to batch_size items, and Python automatically handles the final short chunk. Because the function is a generator, it produces chunks lazily — it does not build a list of all batches in memory, which is ideal for large datasets. Each yield sends one chunk to the caller, and the loop in main prints them in order.

Common mistakes

Returning a list of chunks instead of yielding them, which increases memory usage.
Using `range(len(rows))` instead of `range(0, len(rows), batch_size)`, missing chunks.
Forgetting that Python slices don't raise errors when the end exceeds the list length, so the last chunk may be shorter than `batch_size`.

Variations

Use `itertools.islice` on a general iterator to batch infinite streams.
Use a list comprehension `[rows[i:i+batch_size] for i in range(0, len(rows), batch_size)]` when you need all chunks at once.

Real-world use cases

Inserting thousands of database rows in bulk `executemany` calls, sending one chunk at a time to avoid timeouts.
Paginating through an API response that returns many records, processing each page of results before requesting the next.
Batching data for an ETL pipeline so that machine learning model inference runs on manageable subsets.

Batch Rows in Chunks with a Generator in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Keep learning

Tutorials

Quizzes