How to Split Data into Chunks and Use Generators in Python
Split a list into fixed-size chunks with a list comprehension and square even numbers lazily with a generator expression.
Python code
15 linesdef split_numbers(data, chunk_size):
return [data[i:i + chunk_size] for i in range(0, len(data), chunk_size)]
def square_even_numbers(numbers):
return (n ** 2 for n in numbers if n % 2 == 0)
if __name__ == "__main__":
sample_data = list(range(1, 21))
chunks = split_numbers(sample_data, 5)
print("Chunks:", chunks)
squares = list(square_even_numbers(sample_data))
print("Squared evens:", squares)Output
Chunks: [[1, 2, 3, 4, 5], [6, 7, 8, 9, 10], [11, 12, 13, 14, 15], [16, 17, 18, 19, 20]]
Squared evens: [4, 16, 36, 64, 100, 144, 196, 256, 324, 400]How it works
The split_numbers function uses a list comprehension with range(0, len(data), chunk_size) to generate starting indices, then slices data[i:i + chunk_size] to create each chunk efficiently. The square_even_numbers function returns a generator expression, which computes each square only when iterated, saving memory for large inputs. Converting the generator with list() forces evaluation and produces the final output. This pattern pairs well: comprehensions for immediate materialization, generators for lazy processing.
Common mistakes
- Forgetting that generators are single-use — calling `list()` twice on the same generator yields an empty list the second time.
- Using `range` incorrectly in the chunk loop, e.g., `range(len(data))`, which produces overlapping slices instead of fixed chunks.
- Returning a list comprehension when a generator is needed for memory-heavy data, defeating the lazy benefit.
Variations
- Use `itertools.islice` to chunk an iterator without materializing the full list.
- Rewrite `split_numbers` with a generator yielding chunks one at a time: `yield data[i:i + chunk_size]`.
Real-world use cases
- Batching API requests by splitting a list of IDs into chunks of 100 to respect rate limits.
- Processing large log files line-by-line with a generator to square-matching metrics without loading everything into RAM.
- Grouping database query results into fixed-size pages for paginated report generation.
Sponsored
More from Comprehensions & generators
- Batch Rows in Chunks with a Generator in Python easy
- Build a Generator Pipeline in Python: Filter Then Map medium
- Build a lazy generator to read file lines in Python easy
- Chunk an Iterable into Batches with a Generator in Python easy
- Convert Data in Python with Comprehensions and Generators easy
- Count Data in Python with Comprehensions and Generators easy
Keep learning
Related tutorials and quizzes for this topic.