Big data & Spark
PySpark jobs, partitioning, batch processing, and large-dataset transform patterns.
How to Mock a Parquet partitionBy Sink in Python
Manually write a DataFrame to partitioned Parquet files, mimicking Spark's partitionBy sink behavior without Spark.
import pyarrow as pa
import pyarrow.parquet as pq
from pathlib import Path
import tempfile
import shutil
def mock_partition_by_sink(data, output_dir, partition_cols):
table = pa.Table.from_pandas(data)
schema = table.schema
unique_combos = table.select(partition_cols).to_pylist()
seen = set()
for…How to use foreachBatch with a mock sink in PySpark
Demonstrates using Spark Structured Streaming's foreachBatch sink to capture and verify streaming batches by writing them into a custom mock sink object.
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, lit
class MockSink:
def __init__(self):
self.batches = []
def write_batch(self, batch_df, batch_id):
# Collect batch data as list of dicts for verification
records = batch_df.collect()
self.batches…Browse by section
Each section groups closely related Python snippets.
Big data & Spark — Python code examples
What you will find here
This page collects big data & spark snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.