Read Parquet-Like Columnar CSV Chunks in Python
A Python generator that reads a CSV file column-by-column, yielding dictionary chunks where each key points to a list of values—mirroring how Parquet stores data columnar.
Python code
38 lines```python
import csv
from pathlib import Path
from typing import Iterator, List
def read_parquet_like_columnar(csv_path: str, column_names: List[str], chunk_size: int = 2) -> Iterator[dict]:
"""Read CSV data in columnar chunks, similar to how parquet stores columns."""
csv_file = Path(csv_path)
with csv_file.open(newline='') as f:
reader = csv.DictReader(f)
for name in column_names:
if name not in reader.fieldnames:
raise ValueError(f"Column '{name}' not found in CSV")
with csv_file.open(newline='') as f:
reader = csv.DictReader(f)
chunk = {col: [] for col in column_names}
for i, row in enumerate(reader):
for col in column_names:
chunk[col].append(row[col])
if (i + 1) % chunk_size == 0:
yield chunk
chunk = {col: [] for col in column_names}
if any(chunk[col] for col in column_names):
yield chunk
if __name__ == "__main__":
import tempfile, os
sample_data = "id,name,age\n1,Alice,30\n2,Bob,25\n3,Carol,35\n4,David,40\n5,Eve,28"
temp_dir = tempfile.mkdtemp()
test_path = os.path.join(temp_dir, "sample.csv")
with open(test_path, 'w') as f:
f.write(sample_data)
for chunk in read_parquet_like_columnar(test_path, ["name", "age"], chunk_size=2):
print(chunk)
``
Output
{'name': ['Alice', 'Bob'], 'age': ['30', '25']}
{'name': ['Carol', 'David'], 'age': ['35', '40']}
{'name': ['Eve'], 'age': ['28']}
How it works
The function first validates that all requested column names exist in the CSV header, raising a ValueError early if not. It then reopens the file to iterate through rows, collecting values into a dictionary of lists, one list per column. After every chunk_size rows, it yields the accumulated columnar chunk and resets the dictionary. The final partial chunk is yielded if there are leftover rows, so no data is lost. This pattern is memory-efficient because it streams row by row rather than loading the whole file into memory at once.
Common mistakes
- Forgetting to validate column names before processing, leading to a KeyError mid-iteration
- Not flushing remaining rows if the total row count isn't a multiple of chunk_size
- Assuming the CSV has a header row when it doesn't, causing DictReader to treat the first row as fieldnames
Variations
- Use numpy.genfromtxt or pandas.read_csv for simpler columnar access, but they load everything into memory.
- For very large CSVs, consider using the `csv` module in combination with `itertools.islice` for manual chunking.
Real-world use cases
- Processing huge CSV exports in chunks to feed into a column-oriented database like ClickHouse.
- Batching CSV rows for parallel processing or machine learning feature extraction without loading all data.
- Converting CSV data into a columnar format for efficient column-wise transformations in ETL pipelines.
Sponsored
More from Files & data
- Append a Line to a Log File in Python easy
- Audit File Permissions Across a Project in Python easy
- Automatically Detect Corrupted Files Using SHA-256 Checksums in Python easy
- Automatically Highlight Data Validation Errors Inside Excel Files in Python easy
- Build a Command-Line To-Do List Application with Data Persistence in Python easy
- Build a File Index by Relative Path Hash Map in Python easy
Keep learning
Related tutorials and quizzes for this topic.