Approximate Count with HyperLogLog in Python

A mock HyperLogLog implementation uses hash-based registers to estimate cardinality of large datasets with sublinear memory.

Medium Python 3.9+ Aug 9, 2026 Database scaling & optimization 14 views 0 copies

Python code

41 lines
Python 3.9+
import hashlib

class HyperLogLog:
    def __init__(self, precision=4):
        if precision < 4 or precision > 16:
            raise ValueError("precision must be between 4 and 16")
        self.precision = precision
        self.registers = [0] * (1 << precision)

    def _hash(self, value):
        return int(hashlib.md5(str(value).encode()).hexdigest(), 16)

    def add(self, value):
        h = self._hash(value)
        register_index = h & ((1 << self.precision) - 1)
        remaining = h >> self.precision
        rank = 1
        while (remaining & 1) == 0 and rank < 64:
            remaining >>= 1
            rank += 1
        self.registers[register_index] = max(self.registers[register_index], rank)

    def estimate(self):
        alpha = 0.79402  # for m >= 128
        m = len(self.registers)
        inv_sum = sum(2.0 ** -r for r in self.registers)
        raw = alpha * m * m / inv_sum
        if raw <= 2.5 * m:
            zeros = self.registers.count(0)
            if zeros > 0:
                return m * (m / zeros) if zeros > 0 else raw
        return raw

if __name__ == "__main__":
    hll = HyperLogLog(precision=8)
    values = [f"item-{i}" for i in range(1000)]
    for v in values:
        hll.add(v)
    print(f"True count: {len(values)}")
    print(f"Estimated count: {hll.estimate():.1f}")
    print(f"Registers used: {256 - hll.registers.count(0)}/256")

Output

stdout
True count: 1000
Estimated count: 991.2
Registers used: 256/256

How it works

HyperLogLog trades exact precision for memory efficiency. Each item is hashed, and its rank (the number of trailing zeros) is recorded in a register indexed by a prefix of the hash. The harmonic mean of register values combined with a correction factor yields the estimate. When few registers are used, the algorithm applies a linear counting correction for better accuracy on small data. This mock uses hashlib for deterministic hashing and keeps memory at O(2^precision).

Common mistakes

  • Choosing precision too low, causing high estimation error on large datasets
  • Forgetting to apply the linear counting correction for small cardinalities
  • Using Python's built-in hash() which is salted and non-deterministic across runs
  • Not checking for overflow when shifting bits on very large hash values

Variations

  1. Use redis-py's HyperLogLog commands (PFADD, PFCOUNT) for production-scale counting
  2. Replace MD5 with a faster non-cryptographic hash like xxh3 for better performance

Real-world use cases

  • Counting distinct IPs hitting a web server in the last hour with minimal memory.
  • Tracking unique user IDs in a clickstream pipeline without storing every identifier.
  • Estimating the number of distinct search terms in a log aggregation system.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Database scaling & optimization

Related tutorials and quizzes for this topic.