Chunk an Iterable into Batches with a Generator in Python

Yield fixed-size batches from any iterable lazily using itertools.islice inside a generator function.

Easy Python 3.9+ Aug 9, 2026 Comprehensions & generators 14 views 0 copies

generators iterators itertools chunking lazy-evaluation

Python code

14 lines

Python 3.9+

from itertools import islice

def chunked(iterable, size):
    iterator = iter(iterable)
    while True:
        batch = list(islice(iterator, size))
        if not batch:
            break
        yield batch

if __name__ == "__main__":
    data = range(10)
    for batch in chunked(data, 3):
        print(batch)

Output

stdout

[0, 1, 2]
[3, 4, 5]
[6, 7, 8]
[9]

How it works

The chunked generator wraps the input in an iterator and repeatedly consumes up to size items with islice. Each call to islice advances the same underlying iterator, so the batches do not overlap. When islice returns fewer items than size (or none at all), the loop stops. This approach works with any iterable, not just sequences, and avoids loading the whole input into memory.

Common mistakes

Forgetting that `islice` on a list without `iter()` works, but for other iterables you must wrap – though calling `iter()` is safe for all.
Yielding a generator from `islice` directly instead of converting to a list, which would consume lazily and break when reused.
Not handling an empty input iterable – the `while True` loop should exit without yielding anything.

Variations

Use `yield from iter(lambda: list(islice(iterable, size)), [])` for a one-liner.
Return lists with `list(chunked(data, 3))` for a materialized result.

Real-world use cases

Batching API calls to process a large dataset in manageable pages without loading it all into RAM.
Splitting a log file into fixed-size chunks for parallel processing with multiprocessing or asyncio tasks.
Paginating database query results in chunks to run ETL jobs in a memory-safe way.

Chunk an Iterable into Batches with a Generator in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Keep learning

Tutorials

Quizzes