Big data & Spark
PySpark jobs, partitioning, batch processing, and large-dataset transform patterns.
How to Use Broadcast Variables as Read-Only in PySpark (Mock Example)
Share a lookup dict across Spark executors with a broadcast variable and verify its read-only behavior in a local mock.
from pyspark import SparkContext, SparkConf
def main():
conf = SparkConf().setAppName("BroadcastMock").setMaster("local[2]")
sc = SparkContext(conf=conf)
lookup = {"a": 1, "b": 2, "c": 3}
broadcast_lookup = sc.broadcast(lookup)
data = ["a", "b", "c", "a", "unknown"]
rdd = sc.parallel…Mock RDD in Python: Simulate Spark RDD Lazy Transformations
Simulate Apache Spark RDD behavior in Python with lazy maps, filters, partitions, and a collect action.
import random
def mock_rdd(data, num_slices=2):
"""
A simple simulation of Spark RDD behavior with lazy evaluation,
transformations, and an action.
"""
class SimpleRDD:
def __init__(self, data, num_slices=2):
self.data = data
self.num_slices = num_slices
…Browse by section
Each section groups closely related Python snippets.
Big data & Spark — Python code examples
What you will find here
This page collects big data & spark snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.