Export Data to Parquet & Feather

You’ve just spent hours cleaning, transforming, and reshaping your dataset, and now you need to save the results. The temptation is to write everything to a CSV and move on. But CSV files crush data types, bloat file size, and take forever to load. If you’ve ever opened a 2 GB CSV and watched your laptop beg for mercy, this lesson is for you. Today, you’ll learn how to export results to Parquet and Feather — two modern columnar formats that are faster, smaller, and more type-preserving than CSV, so you can share and reuse your analysis without the pain.

The Problem This Lesson Solves

In real data analysis, the pain is real: CSV is the default export format, and it’s terrible for anything beyond trivial datasets. Parquet and Feather solve this by storing data in a binary, columnar format that preserves dtypes, compresses well, and loads at lightning speed. For example, a 500 MB CSV might compress to 80 MB in Parquet, and reading a Parquet file can be 5–10x faster than reading the equivalent CSV. This lesson shows you exactly how to export results to Parquet and Feather, so you can stop wrestling with CSVs and start working smarter.

Core Concept / Mental Model

Think of CSV as a plain-text inventory list: human-readable, but slow to process and easy to misread (e.g., a date string that gets interpreted as text). Parquet and Feather are like a zip file with a built-in index — they store data in a binary columnar format, which means computations can skip columns you don’t need and load only the rows that matter.

• Parquet is a compressed, columnar storage format widely used in big data (Spark, Hadoop, cloud data lakes). It supports complex nested data and is optimized for storage efficiency.
• Feather is an IPC (InterProcess Communication) format, designed for speed — it writes and reads data nearly as fast as the filesystem will allow, making it ideal for quick transfer between Python and R or for caching intermediate results.

The key insight: format choice matters more than you think. Exporting results to Parquet or Feather isn’t just about saving a file — it’s about preserving your data’s integrity and making future analysis faster.

How It Works Step by Step

Here’s the logical sequence to export a pandas DataFrame to Parquet or Feather:

1. Import pandas and ensure you have the right engine — pandas uses pyarrow or fastparquet as backends for Parquet, and pyarrow for Feather.
2. Call the right methods — for Parquet: df.to_parquet('file.parquet'); for Feather: df.to_feather('file.feather').
3. Check dependencies — if you haven’t installed pyarrow, run pip install pyarrow to use both formats.
4. Verify the output — reload with pd.read_parquet() or pd.read_feather() to ensure data integrity.
5. Consider file size and speed — you can tweak compression (e.g., snappy for Parquet) to balance size vs. write time.

Visual analogy: Saving to CSV is like writing a novel by hand — slow and error-prone; saving to Parquet is like sending a PDF — fast, compact, and ready to read.

Hands-On Walkthrough

Let’s export results to Parquet and Feather with real code. First, make sure you have the required libraries:

pip install pandas pyarrow

Now, create a sample DataFrame and export it to both formats:

import pandas as pd
import numpy as np

# Create a sample DataFrame with mixed dtypes
np.random.seed(42)
df = pd.DataFrame({
    'id': range(1000),
    'category': np.random.choice(['A', 'B', 'C'], size=1000),
    'value': np.random.randn(1000),
    'timestamp': pd.date_range('2024-01-01', periods=1000, freq='h')
})

# Export to Parquet
df.to_parquet('results.parquet', index=False)
print('Parquet file saved.')

# Export to Feather
df.to_feather('results.feather')
print('Feather file saved.')

Now, read them back and verify they’re identical to the original:

# Read back the files
parquet_df = pd.read_parquet('results.parquet')
feather_df = pd.read_feather('results.feather')

# Check equality
print(parquet_df.equals(df))   # True
print(feather_df.equals(df))   # True

# Check dtypes are preserved
print(df.dtypes)
print(parquet_df.dtypes)

Expected output (simplified):

True
True
id                     int64
category              object
timestamp    datetime64[ns]
...

You can also specify compression for Parquet to reduce file size further:

# Use gzip compression for smaller files (but slower)
df.to_parquet('results_compressed.parquet', compression='gzip')

Pro tip: Always set index=False when exporting to Parquet if you don’t need the index — it keeps the format clean and avoids surprises when reading back.

Compare Options / When to Choose What

Not every format fits every job. Here’s a practical comparison:

When to choose what: Use Parquet when you’re storing data for later analysis in a distributed system or cloud. Use Feather when you need to move data between processes or notebooks quickly and want maximum speed. CSV is still fine for sharing with non-technical users, but for everything else, prefer binary formats.

Troubleshooting & Edge Cases

• ImportError: missing pyarrow — Install it: pip install pyarrow. If you’re using fastparquet, install that instead.
• TypeError: object of type ... is not supported — Some pandas dtypes (like category with unknown categories) may not round-trip. Convert to string or numeric before export if you encounter this.
• Feather doesn’t support nested data — Parquet handles nested columns; Feather is for flat data. If you have dicts in columns, stick to Parquet.
• **to_parquet with index=False — If you forget to set it, the index becomes a column on read-back, which can cause schema mismatches. Always be explicit.
• Large files and memory — Loading a huge Parquet file may still be heavy. Use columns parameter to load only certain columns:

# Read only a few columns
subset = pd.read_parquet('results.parquet', columns=['id', 'value'])

What You Learned & What’s Next

You now know how to export results to Parquet and Feather using pandas, preserving data types and making your workflows faster. You’ve mastered the core concepts, step-by-step coding, and troubleshooting. These binary formats are essential for any serious data analysis, especially when dealing with large datasets.

Next, you’ll learn how to import data from API responses — turning JSON into tidy DataFrames, so you can pull live data into your analysis pipelines. With your new export skills, you’ll be ready to build efficient, reproducible workflows from data collection to final output.