Read Parquet-Like Columnar CSV Chunks in Python

A Python generator that reads a CSV file column-by-column, yielding dictionary chunks where each key points to a list of values—mirroring how Parquet stores data columnar.

Medium Python 3.9+ Aug 9, 2026 Files & data 13 views 0 copies

Python code

38 lines
Python 3.9+
```python
import csv
from pathlib import Path
from typing import Iterator, List

def read_parquet_like_columnar(csv_path: str, column_names: List[str], chunk_size: int = 2) -> Iterator[dict]:
    """Read CSV data in columnar chunks, similar to how parquet stores columns."""
    csv_file = Path(csv_path)
    with csv_file.open(newline='') as f:
        reader = csv.DictReader(f)
        for name in column_names:
            if name not in reader.fieldnames:
                raise ValueError(f"Column '{name}' not found in CSV")

    with csv_file.open(newline='') as f:
        reader = csv.DictReader(f)
        chunk = {col: [] for col in column_names}
        for i, row in enumerate(reader):
            for col in column_names:
                chunk[col].append(row[col])
            if (i + 1) % chunk_size == 0:
                yield chunk
                chunk = {col: [] for col in column_names}
        if any(chunk[col] for col in column_names):
            yield chunk

if __name__ == "__main__":
    import tempfile, os

    sample_data = "id,name,age\n1,Alice,30\n2,Bob,25\n3,Carol,35\n4,David,40\n5,Eve,28"
    temp_dir = tempfile.mkdtemp()
    test_path = os.path.join(temp_dir, "sample.csv")
    with open(test_path, 'w') as f:
        f.write(sample_data)

    for chunk in read_parquet_like_columnar(test_path, ["name", "age"], chunk_size=2):
        print(chunk)
``

Output

stdout
{'name': ['Alice', 'Bob'], 'age': ['30', '25']}
{'name': ['Carol', 'David'], 'age': ['35', '40']}
{'name': ['Eve'], 'age': ['28']}

How it works

The function first validates that all requested column names exist in the CSV header, raising a ValueError early if not. It then reopens the file to iterate through rows, collecting values into a dictionary of lists, one list per column. After every chunk_size rows, it yields the accumulated columnar chunk and resets the dictionary. The final partial chunk is yielded if there are leftover rows, so no data is lost. This pattern is memory-efficient because it streams row by row rather than loading the whole file into memory at once.

Common mistakes

  • Forgetting to validate column names before processing, leading to a KeyError mid-iteration
  • Not flushing remaining rows if the total row count isn't a multiple of chunk_size
  • Assuming the CSV has a header row when it doesn't, causing DictReader to treat the first row as fieldnames

Variations

  1. Use numpy.genfromtxt or pandas.read_csv for simpler columnar access, but they load everything into memory.
  2. For very large CSVs, consider using the `csv` module in combination with `itertools.islice` for manual chunking.

Real-world use cases

  • Processing huge CSV exports in chunks to feed into a column-oriented database like ClickHouse.
  • Batching CSV rows for parallel processing or machine learning feature extraction without loading all data.
  • Converting CSV data into a columnar format for efficient column-wise transformations in ETL pipelines.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Files & data

Related tutorials and quizzes for this topic.