Big data & Spark
PySpark jobs, partitioning, batch processing, and large-dataset transform patterns.
Bloom Filter Join Mock in Python
A mock hash join that uses a Bloom filter to pre-filter one table before performing an exact match, reducing the number of comparisons in large dataset joins.
import hashlib
import random
import string
class BloomFilter:
def __init__(self, size: int = 200, num_hashes: int = 3):
self.bits = [False] * size
self.size = size
self.num_hashes = num_hashes
def _hashes(self, item: str):
result = []
for seed in range(self.num_hashes…Skew Join Salting Key in Python (Demo)
Demonstrates skew join salting by expanding a smaller side with salt keys and matching rows on the larger side via random salt assignment.
import random
def skew_join_salting_key(left_df, right_df, salt_range=4):
"""
Demonstrates skew join salting: expand the smaller side with salt keys,
then attach a salt key to each row on the larger side.
Returns a list of (left, right, salt) tuples.
"""
skewed_left = []
for row in left_d…Browse by section
Each section groups closely related Python snippets.
Big data & Spark — Python code examples
What you will find here
This page collects big data & spark snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.