Memory efficient map over large file in Python

A generator-based streaming map that processes a large file line by line without loading the whole file into memory.

Easy Python 3.9+ Aug 9, 2026 Comprehensions & generators 13 views 0 copies

generator file-io streaming memory

Python code

25 lines

Python 3.9+

import sys

def process_lines(file_path):
    """Memory-efficient map over a large file: yields processed lines."""
    with open(file_path, 'r') as f:
        for line in f:
            # Example mapping: strip whitespace and uppercase
            yield line.strip().upper()

if __name__ == "__main__":
    # Use a small inline example; works for any large file path
    import tempfile
    import os

    # Create a sample file
    with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt') as tmp:
        tmp.write("hello world\npython rocks\nmemory efficient\n")
        tmp_path = tmp.name

    # Process line by line (streaming)
    for processed in process_lines(tmp_path):
        print(processed)

    # Cleanup
    os.unlink(tmp_path)

Output

stdout

HELLO WORLD
PYTHON ROCKS
MEMORY EFFICIENT

How it works

This code defines a generator function process_lines that opens the file and iterates over each line lazily. Instead of reading the entire file into a list, it yields each processed line one at a time, so memory usage stays constant regardless of file size. The with statement ensures the file is closed properly. The example creates a temporary file to demonstrate the output.

Common mistakes

Calling `readlines()` or `read()` inside the loop, which loads the whole file into memory.
Forgetting to close the file if using a manual open without `with`.
Assuming the generator processes everything at once; it only yields when iterated.

Variations

Use a list comprehension with `readlines()` for small files: `[line.strip().upper() for line in open('file.txt')]`.
Use `map` with a generator expression: `(line.strip().upper() for line in open('file.txt'))`.

Real-world use cases

Processing multi-gigabyte log files to extract key fields while keeping memory under control.
Streaming large CSV or TSV files to transform rows for a data pipeline without OOM.
Reading big text exports from legacy systems line by line to normalize formats.

Memory efficient map over large file in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Keep learning

Tutorials

Quizzes