Approximate Count with HyperLogLog in Python
A mock HyperLogLog implementation uses hash-based registers to estimate cardinality of large datasets with sublinear memory.
Python code
41 linesimport hashlib
class HyperLogLog:
def __init__(self, precision=4):
if precision < 4 or precision > 16:
raise ValueError("precision must be between 4 and 16")
self.precision = precision
self.registers = [0] * (1 << precision)
def _hash(self, value):
return int(hashlib.md5(str(value).encode()).hexdigest(), 16)
def add(self, value):
h = self._hash(value)
register_index = h & ((1 << self.precision) - 1)
remaining = h >> self.precision
rank = 1
while (remaining & 1) == 0 and rank < 64:
remaining >>= 1
rank += 1
self.registers[register_index] = max(self.registers[register_index], rank)
def estimate(self):
alpha = 0.79402 # for m >= 128
m = len(self.registers)
inv_sum = sum(2.0 ** -r for r in self.registers)
raw = alpha * m * m / inv_sum
if raw <= 2.5 * m:
zeros = self.registers.count(0)
if zeros > 0:
return m * (m / zeros) if zeros > 0 else raw
return raw
if __name__ == "__main__":
hll = HyperLogLog(precision=8)
values = [f"item-{i}" for i in range(1000)]
for v in values:
hll.add(v)
print(f"True count: {len(values)}")
print(f"Estimated count: {hll.estimate():.1f}")
print(f"Registers used: {256 - hll.registers.count(0)}/256")
Output
True count: 1000
Estimated count: 991.2
Registers used: 256/256
How it works
HyperLogLog trades exact precision for memory efficiency. Each item is hashed, and its rank (the number of trailing zeros) is recorded in a register indexed by a prefix of the hash. The harmonic mean of register values combined with a correction factor yields the estimate. When few registers are used, the algorithm applies a linear counting correction for better accuracy on small data. This mock uses hashlib for deterministic hashing and keeps memory at O(2^precision).
Common mistakes
- Choosing precision too low, causing high estimation error on large datasets
- Forgetting to apply the linear counting correction for small cardinalities
- Using Python's built-in hash() which is salted and non-deterministic across runs
- Not checking for overflow when shifting bits on very large hash values
Variations
- Use redis-py's HyperLogLog commands (PFADD, PFCOUNT) for production-scale counting
- Replace MD5 with a faster non-cryptographic hash like xxh3 for better performance
Real-world use cases
- Counting distinct IPs hitting a web server in the last hour with minimal memory.
- Tracking unique user IDs in a clickstream pipeline without storing every identifier.
- Estimating the number of distinct search terms in a log aggregation system.
Sponsored
More from Database scaling & optimization
- B-Tree Insert and In-Order Traversal in Python hard
- Broadcast a Small Reference Table in Python easy
- Build a Full Text Search Index in Python medium
- Build a Partial Index Mock in Python for Database Filtering easy
- Composite index leftmost prefix in Python medium
- Consistent Hashing with Virtual Buckets in Python medium
Keep learning
Related tutorials and quizzes for this topic.