Build a Generator Pipeline in Python: Filter Then Map

Create a lazy data pipeline by chaining generator functions that read, filter, map, and write data step by step.

Medium Python 3.9+ Aug 9, 2026 Comprehensions & generators 12 views 0 copies

generators pipeline lazy-evaluation filter map

Python code

20 lines

Python 3.9+

def read_data():
    return ["a", "bb", "ccc", "dd", "eeeee", "f"]


def filter_short(words):
    return (word for word in words if len(word) >= 2)


def map_to_upper(words):
    return (word.upper() for word in words)


def write_data(words):
    for word in words:
        print(word)


if __name__ == "__main__":
    pipeline = map_to_upper(filter_short(read_data()))
    write_data(pipeline)

Output

stdout

BB
CCC
DD
EEEEE

How it works

Each function returns a generator expression, which produces values lazily rather than building an entire list in memory. Chaining them with map_to_upper(filter_short(read_data())) connects the pipeline without intermediate storage. The write_data loop pulls one item at a time through the chain, so memory usage stays constant no matter how large the source. This pattern keeps each step small, testable, and easy to reorder or extend.

Common mistakes

Using regular function returns with lists instead of generator expressions
Forgetting that generators can only be iterated once
Confusing generator expressions with list comprehensions (extra brackets)

Variations

Use the built-in `filter()` and `map()` functions instead of custom generators
Add a `transform` step that parses or normalizes items between filter and map

Real-world use cases

Streaming log lines from a file, filtering errors, and transforming them for output
Processing massive CSV rows without loading the whole file into memory
Chaining API responses into a cleanup step then a serialization step

Build a Generator Pipeline in Python: Filter Then Map

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Keep learning

Tutorials

Quizzes