Big data & Spark
PySpark jobs, partitioning, batch processing, and large-dataset transform patterns.
Bloom Filter Join Mock in Python
A mock hash join that uses a Bloom filter to pre-filter one table before performing an exact match, reducing the number of comparisons in large dataset joins.
import hashlib
import random
import string
class BloomFilter:
def __init__(self, size: int = 200, num_hashes: int = 3):
self.bits = [False] * size
self.size = size
self.num_hashes = num_hashes
def _hashes(self, item: str):
result = []
for seed in range(self.num_hashes…How to Mock a Hash Join on Large and Small Tables in Python
This code efficiently joins a large dataset (1000 rows) with a small lookup table (20 rows) by building a dictionary hash lookup, mimicking a hash join strategy used in big data systems.
import random
from pprint import pprint
# Large table: 1000 rows (id, group_id, value)
large = [{"id": i, "group_id": random.randint(1, 20), "value": random.random() * 100} for i in range(1000)]
# Small table: 20 rows (group_id, label)
small = [{"group_id": g, "label": f"Group-{g}"} for g in range(1, 21)]
# Mock a …HyperLogLog Cardinality Estimation in Python
A small HyperLogLog implementation using MD5 hashing and 256 registers to estimate the number of unique items in a large stream with fixed memory.
import hashlib
import math
class HyperLogLog:
def __init__(self, b=8):
self.b = b
self.m = 1 << b
self.registers = [0] * self.m
self.alpha = 0.7213 / (1 + 1.079 / self.m)
def add(self, item):
h = int(hashlib.md5(str(item).encode()).hexdigest(), 16)
idx = h & (s…Partition Data by Hash Key Mod N in Python
Returns a partition index for a string key by hashing it with MD5 and taking modulo N, then groups sample keys into partitions.
import hashlib
def partition_key(key: str, num_partitions: int) -> int:
"""Return partition index for key using MD5 hash mod N."""
digest = hashlib.md5(key.encode()).hexdigest()
return int(digest, 16) % num_partitions
if __name__ == "__main__":
keys = ["alice", "bob", "carol", "dave", "eve"]
nu…Browse by section
Each section groups closely related Python snippets.
Big data & Spark — Python code examples
What you will find here
This page collects big data & spark snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.