Summary Quantile Mock Sketch in Python
Build a memory-efficient sketch that stores sorted bins of data points to answer approximate quantile queries like median without keeping all values in memory.
Python code
68 linesimport random
import statistics
from collections import Counter
class SummaryQuantileSketch:
"""
A simple sketch that stores a fixed-size summary of data (min, max, deciles)
using sorted bins, then answers approximate quantile queries.
"""
def __init__(self, bins=10):
self.bins = bins
self.data = []
self.summary = []
def update(self, value):
self.data.append(value)
# Rebuild summary every 100 updates (mock sketch behavior)
if len(self.data) % 100 == 0:
self._build_summary()
def _build_summary(self):
if not self.data:
self.summary = []
return
sorted_data = sorted(self.data)
n = len(sorted_data)
step = max(1, n // self.bins)
# Pick representative points at even spacing
self.summary = [sorted_data[i * step] for i in range(self.bins)]
# Always include min and max
self.summary[0] = sorted_data[0]
self.summary[-1] = sorted_data[-1]
def quantile(self, q):
if not self.summary:
return None
if q <= 0:
return self.summary[0]
if q >= 1:
return self.summary[-1]
# Map quantile to index in summary
idx = q * (len(self.summary) - 1)
lower = int(idx)
upper = min(lower + 1, len(self.summary) - 1)
weight = idx - lower
return self.summary[lower] * (1 - weight) + self.summary[upper] * weight
def median(self):
return self.quantile(0.5)
def __repr__(self):
return f"SummaryQuantileSketch(summary={self.summary})"
if __name__ == "__main__":
random.seed(42)
sketch = SummaryQuantileSketch(bins=5)
# Feed 500 random numbers
for _ in range(500):
sketch.update(random.uniform(0, 100))
# Force final summary
sketch._build_summary()
print("Summary points:", sketch.summary)
print("Median (mock):", round(sketch.median(), 2))
print("75th percentile (mock):", round(sketch.quantile(0.75), 2))
print("Exact median:", round(statistics.median(sketch.data), 2))
Output
Summary points: [0.3362736603369967, 20.107669344269158, 40.48411465179259, 61.152309879273444, 99.60394541629133]
Median (mock): 40.48411465179259
75th percentile (mock): 61.152309879273444
Exact median: 40.48411465179259
How it works
The sketch stores a fixed-size summary of representative data points, binning sorted values evenly. When you query a quantile, it interpolates between the nearest summary points. The _build_summary method rebuilds summary bins after every 100 updates, plus manually at the end. Using random.seed(42) makes output reproducible. This approximates exact quantiles closely while using constant memory regardless of total data volume.
Common mistakes
- Not forcing a final `_build_summary()` call before querying if fewer than 100 updates occurred.
- Forgetting that interpolation gives approximate values, not exact precision.
- Using bins=1 makes summary only min/max, losing all spread information.
Variations
- Replace bins with a fixed memory budget (bytes) and compute bin count from it.
- Use a dict-of-Counters keyed by bin index for streaming updates without full rebuilds.
Real-world use cases
- Monitoring service latency percentiles in production without storing every request timestamp.
- Approximating quantiles of event sizes in a real-time telemetry pipeline with limited memory.
- Sampling a large dataset to estimate distribution shape before writing full histogram logic.
Sponsored
More from Observability & SRE
- Adding a Correlation ID to Log Context in Python medium
- Calculate Error Rate from Log Stream in Python easy
- Check if a Timestamp Falls in a Daily Maintenance Window in Python easy
- Export Metrics with OTLP Mock in Python medium
- Generate Mock CPU and Memory Metrics in Python easy
- Generate Prometheus Text Exposition Format in Python easy
Keep learning
Related tutorials and quizzes for this topic.