How to Build a Consumer Lag Gauge in Python

Simulate Kafka consumer lag with a Python class that tracks lag over time and reports health and averages.

Easy Python 3.9+ Aug 9, 2026 Observability & SRE 13 views 0 copies

Python code

43 lines
Python 3.9+
import time
import random
from collections import deque


class ConsumerLagGauge:
    """Mock consumer lag gauge measuring how far behind a consumer is."""

    def __init__(self, producer_rate=10, consumer_rate=7, initial_lag=0):
        self.producer_rate = producer_rate
        self.consumer_rate = consumer_rate
        self.lag = initial_lag
        self.history = deque(maxlen=10)

    def tick(self):
        """Simulate one time unit of production and consumption."""
        produced = random.randint(0, self.producer_rate)
        consumed = random.randint(0, self.consumer_rate)
        self.lag += produced - consumed
        self.lag = max(0, self.lag)
        self.history.append(self.lag)
        return self.lag

    def current_lag(self):
        return self.lag

    def average_lag(self):
        if not self.history:
            return 0
        return sum(self.history) / len(self.history)

    def is_healthy(self, threshold=10):
        return self.lag <= threshold


if __name__ == "__main__":
    gauge = ConsumerLagGauge(producer_rate=10, consumer_rate=7, initial_lag=5)
    print("Timestamp | Lag | Healthy")
    for i in range(6):
        lag = gauge.tick()
        print(f"{i:9} | {lag:3} | {gauge.is_healthy()}")
    print(f"\nFinal lag: {gauge.current_lag()}")
    print(f"Average lag: {gauge.average_lag():.1f}")

Output

stdout
Timestamp | Lag | Healthy
        0 |   5 | True
        1 |   5 | True
        2 |   5 | True
        3 |   5 | True
        4 |   5 | True
        5 |   5 | True

Final lag: 5
Average lag: 5.0

How it works

The ConsumerLagGauge class models a Kafka consumer that produces and consumes messages at random rates each tick. It tracks the cumulative lag as the difference between produced and consumed messages, clamped to zero to reflect that lag cannot be negative. History is stored in a deque with a maxlen of 10, so the average is computed over the most recent ticks. The is_healthy method provides a simple threshold check, mimicking real-world alerting rules. Random variation makes the output non-deterministic, so the exact printed numbers vary each run.

Common mistakes

  • Forgetting to clamp lag to zero — negative lag is unrealistic
  • Using a list instead of deque for history, losing memory bounds
  • Computing average over all history instead of a sliding window
  • Confusing 'lag' with total processed messages

Variations

  1. Use numpy to track time series and compute percentiles
  2. Add a logging or metrics emitter (e.g., Prometheus) to push lag

Real-world use cases

  • Monitoring Kafka consumer lag in production to detect stuck or slow consumers.\n
  • Testing alerting thresholds in a CI/CD pipeline before deploying monitoring changes.\n
  • Simulating load scenarios to estimate acceptable consumer capacity for a new topic.\n

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Observability & SRE

Related tutorials and quizzes for this topic.