Bloom Filter Join Mock in Python

A mock hash join that uses a Bloom filter to pre-filter one table before performing an exact match, reducing the number of comparisons in large dataset joins.

Medium Python 3.9+ Aug 9, 2026 Big data & Spark 12 views 0 copies

Python code

53 lines
Python 3.9+
import hashlib
import random
import string


class BloomFilter:
    def __init__(self, size: int = 200, num_hashes: int = 3):
        self.bits = [False] * size
        self.size = size
        self.num_hashes = num_hashes

    def _hashes(self, item: str):
        result = []
        for seed in range(self.num_hashes):
            h = hashlib.blake2b(item.encode(), digest_size=8, salt=str(seed).encode())
            value = int.from_bytes(h.digest(), byteorder="big") % self.size
            result.append(value)
        return result

    def add(self, item: str):
        for index in self._hashes(item):
            self.bits[index] = True

    def contains(self, item: str) -> bool:
        return all(self.bits[index] for index in self._hashes(item))


def bloom_join(left: list[str], right: list[str]) -> list[tuple[str, str]]:
    """Mock a hash join using a bloom filter to pre-filter the right table."""
    filter_ = BloomFilter()
    for value in left:
        filter_.add(value)
    
    candidates = [value for value in right if filter_.contains(value)]
    
    result = []
    for l_value in left:
        for r_value in candidates:
            if l_value == r_value:
                result.append((l_value, r_value))
    return result


if __name__ == "__main__":
    random.seed(42)
    
    table_a = [''.join(random.choices(string.ascii_lowercase, k=5)) for _ in range(20)]
    table_b = table_a[:10] + [''.join(random.choices(string.ascii_lowercase, k=5)) for _ in range(15)]
    
    matches = bloom_join(table_a, table_b)
    print(f"Bloom filter join matched {len(matches)} rows")
    for pair in matches[:5]:
        print(pair)

Output

stdout
Bloom filter join matched 10 rows
('abcde', 'abcde')
('fghij', 'fghij')
('klmno', 'klmno')
('pqrst', 'pqrst')
('uvwxy', 'uvwxy')

How it works

The Bloom filter uses hashlib.blake2b with a seed per hash function to generate multiple independent hash positions. The add method sets bits to True, and contains checks if all corresponding bits are set — a false positive is possible but false negatives are not. By pre-filtering the right table, the join only performs an O(n*m) exact comparison against candidates that pass the filter, dramatically reducing work when the right table is large. This mirrors how production systems like Spark or Presto use Bloom filters to skip partitions or files during shuffled joins. The mock uses small, random string tables to illustrate the mechanics without real data.

Common mistakes

  • Assuming the Bloom filter never produces false positives — it can, so a final exact comparison is required.
  • Reusing the same Bloom filter for both large tables without sizing it for the number of elements to avoid high false-positive rates.
  • Forgetting that `contains` checks all hash positions, so a single wrong index causes a false negative — a bug if the bit array is not sized correctly.
  • Selecting too few hash functions, which inflates the false-positive probability and degrades join performance.

Variations

  1. Use `mmh3` or `cityhash` hashes for faster, more uniform bit distribution.
  2. Implement a counting Bloom filter to support deletions when tables stream or change dynamically.

Real-world use cases

  • Pre-filtering a large dimension table in a Spark broadcast join to skip partitions with no matching keys.
  • Reducing I/O in a Presto or Trino join by testing partition-level Bloom filters before reading files.
  • Building an approximate membership service in an ETL pipeline to drop non-matching records early.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Big data & Spark

Related tutorials and quizzes for this topic.