Bloom Filter Join Mock in Python
A mock hash join that uses a Bloom filter to pre-filter one table before performing an exact match, reducing the number of comparisons in large dataset joins.
Python code
53 linesimport hashlib
import random
import string
class BloomFilter:
def __init__(self, size: int = 200, num_hashes: int = 3):
self.bits = [False] * size
self.size = size
self.num_hashes = num_hashes
def _hashes(self, item: str):
result = []
for seed in range(self.num_hashes):
h = hashlib.blake2b(item.encode(), digest_size=8, salt=str(seed).encode())
value = int.from_bytes(h.digest(), byteorder="big") % self.size
result.append(value)
return result
def add(self, item: str):
for index in self._hashes(item):
self.bits[index] = True
def contains(self, item: str) -> bool:
return all(self.bits[index] for index in self._hashes(item))
def bloom_join(left: list[str], right: list[str]) -> list[tuple[str, str]]:
"""Mock a hash join using a bloom filter to pre-filter the right table."""
filter_ = BloomFilter()
for value in left:
filter_.add(value)
candidates = [value for value in right if filter_.contains(value)]
result = []
for l_value in left:
for r_value in candidates:
if l_value == r_value:
result.append((l_value, r_value))
return result
if __name__ == "__main__":
random.seed(42)
table_a = [''.join(random.choices(string.ascii_lowercase, k=5)) for _ in range(20)]
table_b = table_a[:10] + [''.join(random.choices(string.ascii_lowercase, k=5)) for _ in range(15)]
matches = bloom_join(table_a, table_b)
print(f"Bloom filter join matched {len(matches)} rows")
for pair in matches[:5]:
print(pair)
Output
Bloom filter join matched 10 rows
('abcde', 'abcde')
('fghij', 'fghij')
('klmno', 'klmno')
('pqrst', 'pqrst')
('uvwxy', 'uvwxy')
How it works
The Bloom filter uses hashlib.blake2b with a seed per hash function to generate multiple independent hash positions. The add method sets bits to True, and contains checks if all corresponding bits are set — a false positive is possible but false negatives are not. By pre-filtering the right table, the join only performs an O(n*m) exact comparison against candidates that pass the filter, dramatically reducing work when the right table is large. This mirrors how production systems like Spark or Presto use Bloom filters to skip partitions or files during shuffled joins. The mock uses small, random string tables to illustrate the mechanics without real data.
Common mistakes
- Assuming the Bloom filter never produces false positives — it can, so a final exact comparison is required.
- Reusing the same Bloom filter for both large tables without sizing it for the number of elements to avoid high false-positive rates.
- Forgetting that `contains` checks all hash positions, so a single wrong index causes a false negative — a bug if the bit array is not sized correctly.
- Selecting too few hash functions, which inflates the false-positive probability and degrades join performance.
Variations
- Use `mmh3` or `cityhash` hashes for faster, more uniform bit distribution.
- Implement a counting Bloom filter to support deletions when tables stream or change dynamically.
Real-world use cases
- Pre-filtering a large dimension table in a Spark broadcast join to skip partitions with no matching keys.
- Reducing I/O in a Presto or Trino join by testing partition-level Bloom filters before reading files.
- Building an approximate membership service in an ETL pipeline to drop non-matching records early.
Sponsored
More from Big data & Spark
- Accumulators Global Counter Mock in Python medium
- Approximate Distinct Count in Python with HyperLogLog medium
- Cache persist MEMORY_ONLY mock in Python easy
- Compaction Small Files Mock in Python easy
- Delta Lake ACID Transaction Log Mock in Python medium
- How to Broadcast a Small Lookup Table in Python easy
Keep learning
Related tutorials and quizzes for this topic.