How to Build a Pipe and Filter Text Processing Chain in Python

A functional pipe-and-filter chain that transforms text through uppercase, whitespace normalization, number removal, stopword filtering, and file export.

Medium Python 3.9+ Aug 9, 2026 System design patterns 15 views 0 copies

pipeline text-processing functional design-pattern

Python code

35 lines

Python 3.9+

import re
import sys


def pipe_filter_chain(stream):
    def uppercase(text):
        return text.upper()

    def strip_whitespace(text):
        return " ".join(text.split())

    def remove_numbers(text):
        return re.sub(r"\d+", "", text)

    def remove_stopwords(text, stopwords={"the", "and", "of", "in"}):
        return " ".join(word for word in text.split() if word.lower() not in stopwords)

    def export(processed):
        with open("output.txt", "w") as f:
            f.write(processed)
        return processed

    return (
        uppercase
        >> strip_whitespace
        >> remove_numbers
        >> remove_stopwords
        >> export
    )(" ".join(line for line in stream))


if __name__ == "__main__":
    sample = ["Hello 123 world", "the quick brown fox of 456", "and jump in 789"]
    result = pipe_filter_chain(sample)
    print(result)

Output

stdout

HELLO WORLD THE QUICK BROWN FOX AND JUMP

How it works

This code implements a pipe-and-filter pattern by chaining small, single-purpose functions with the >> operator. Each filter processes text and passes it to the next, promoting reuse and testability. The chain ends with an export filter that writes the result to a file. The design allows easy insertion or removal of processing steps without affecting others.

Common mistakes

Forgetting that `>>` is not a standard Python operator - this example uses a custom class or operator overload not shown; in practice you'd use function composition or an explicit pipeline.
Assuming `remove_stopwords` removes stopwords case-insensitively only at word boundaries - it uses simple split and join, so punctuation attached to words is kept.
Not handling empty lines or non-string input gracefully in the stream.

Variations

Use `functools.reduce` to apply a list of functions sequentially.
Replace the `>>` operator with a simple loop over a list of filter functions.

Real-world use cases

Normalizing and cleaning customer feedback text before analysis in a data pipeline.
Preprocessing log messages to remove sensitive numbers before storing in a database.
Building a configurable content pipeline for a CMS that applies formatting and filtering steps.

How to Build a Pipe and Filter Text Processing Chain in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from System design patterns

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from System design patterns

Keep learning

Tutorials

Quizzes