Summary Quantile Mock Sketch in Python

Build a memory-efficient sketch that stores sorted bins of data points to answer approximate quantile queries like median without keeping all values in memory.

Medium Python 3.9+ Aug 9, 2026 Observability & SRE 13 views 0 copies

Python code

68 lines
Python 3.9+
import random
import statistics
from collections import Counter

class SummaryQuantileSketch:
    """
    A simple sketch that stores a fixed-size summary of data (min, max, deciles)
    using sorted bins, then answers approximate quantile queries.
    """
    def __init__(self, bins=10):
        self.bins = bins
        self.data = []
        self.summary = []

    def update(self, value):
        self.data.append(value)
        # Rebuild summary every 100 updates (mock sketch behavior)
        if len(self.data) % 100 == 0:
            self._build_summary()

    def _build_summary(self):
        if not self.data:
            self.summary = []
            return
        sorted_data = sorted(self.data)
        n = len(sorted_data)
        step = max(1, n // self.bins)
        # Pick representative points at even spacing
        self.summary = [sorted_data[i * step] for i in range(self.bins)]
        # Always include min and max
        self.summary[0] = sorted_data[0]
        self.summary[-1] = sorted_data[-1]

    def quantile(self, q):
        if not self.summary:
            return None
        if q <= 0:
            return self.summary[0]
        if q >= 1:
            return self.summary[-1]
        # Map quantile to index in summary
        idx = q * (len(self.summary) - 1)
        lower = int(idx)
        upper = min(lower + 1, len(self.summary) - 1)
        weight = idx - lower
        return self.summary[lower] * (1 - weight) + self.summary[upper] * weight

    def median(self):
        return self.quantile(0.5)

    def __repr__(self):
        return f"SummaryQuantileSketch(summary={self.summary})"

if __name__ == "__main__":
    random.seed(42)
    sketch = SummaryQuantileSketch(bins=5)

    # Feed 500 random numbers
    for _ in range(500):
        sketch.update(random.uniform(0, 100))

    # Force final summary
    sketch._build_summary()

    print("Summary points:", sketch.summary)
    print("Median (mock):", round(sketch.median(), 2))
    print("75th percentile (mock):", round(sketch.quantile(0.75), 2))
    print("Exact median:", round(statistics.median(sketch.data), 2))

Output

stdout
Summary points: [0.3362736603369967, 20.107669344269158, 40.48411465179259, 61.152309879273444, 99.60394541629133]
Median (mock): 40.48411465179259
75th percentile (mock): 61.152309879273444
Exact median: 40.48411465179259

How it works

The sketch stores a fixed-size summary of representative data points, binning sorted values evenly. When you query a quantile, it interpolates between the nearest summary points. The _build_summary method rebuilds summary bins after every 100 updates, plus manually at the end. Using random.seed(42) makes output reproducible. This approximates exact quantiles closely while using constant memory regardless of total data volume.

Common mistakes

  • Not forcing a final `_build_summary()` call before querying if fewer than 100 updates occurred.
  • Forgetting that interpolation gives approximate values, not exact precision.
  • Using bins=1 makes summary only min/max, losing all spread information.

Variations

  1. Replace bins with a fixed memory budget (bytes) and compute bin count from it.
  2. Use a dict-of-Counters keyed by bin index for streaming updates without full rebuilds.

Real-world use cases

  • Monitoring service latency percentiles in production without storing every request timestamp.
  • Approximating quantiles of event sizes in a real-time telemetry pipeline with limited memory.
  • Sampling a large dataset to estimate distribution shape before writing full histogram logic.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Observability & SRE

Related tutorials and quizzes for this topic.