Big data & Spark
PySpark jobs, partitioning, batch processing, and large-dataset transform patterns.
Accumulators Global Counter Mock in Python
Shows an accumulator-style global counter with a mock patch to control its value in tests.
import unittest
from unittest.mock import patch
# Module-level global counter accumulator
counter = 0
def increment(by=1):
"""Increment the global counter in place (accumulator pattern)."""
global counter
counter += by
return counter
def reset():
"""Reset the counter to zero."""
global count…How to Broadcast a Small Lookup Table in Python
Simulates broadcasting a small lookup table by iterating key-value pairs and emitting packed rows to subscribers with deterministic output.
import random
# Generate a deterministic mock broadcast of a small lookup table
# with 5 keys and random integer values (seeded for reproducibility)
data = {
"sensor_a": 22,
"sensor_b": 87,
"sensor_c": 43,
"sensor_d": 65,
"sensor_e": 31,
}
# Simulate a broadcast to subscribers by iterating and p…How to Implement collect_list in Python
Group rows by a key and collect all corresponding values into a list — a pure-Python mock of Spark's collect_list aggregation.
from collections import defaultdict
def collect_list(rows, key_field, value_field):
grouped = defaultdict(list)
for row in rows:
grouped[row[key_field]].append(row[value_field])
return dict(grouped)
if __name__ == "__main__":
data = [
{"dept": "sales", "emp": "alice"},
{"dept"…How to Pivot and Group Aggregate in Python
Group records by a key, collect values, and apply an aggregate function (like sum) to build a pivot-style summary dictionary.
from collections import defaultdict
def pivot_group_aggregate(records, group_key, value_key, agg_func):
groups = defaultdict(list)
for record in records:
groups[record[group_key]].append(record[value_key])
return {key: agg_func(values) for key, values in groups.items()}
if __name__ == "__main__":…Sliding Window Streaming Mock in Python
A simple Python class that maintains a sliding window of recent streaming values and computes the running average.
import time
import random
class StreamingMock:
"""Produces a stream of numbers using a sliding window."""
def __init__(self, window_size=5):
self.window = []
self.window_size = window_size
def push(self, value):
"""Add a value, sliding the window forward."""
s…Browse by section
Each section groups closely related Python snippets.
Big data & Spark — Python code examples
What you will find here
This page collects big data & spark snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.