How to Build a Consumer Lag Gauge in Python
Simulate Kafka consumer lag with a Python class that tracks lag over time and reports health and averages.
Python code
43 linesimport time
import random
from collections import deque
class ConsumerLagGauge:
"""Mock consumer lag gauge measuring how far behind a consumer is."""
def __init__(self, producer_rate=10, consumer_rate=7, initial_lag=0):
self.producer_rate = producer_rate
self.consumer_rate = consumer_rate
self.lag = initial_lag
self.history = deque(maxlen=10)
def tick(self):
"""Simulate one time unit of production and consumption."""
produced = random.randint(0, self.producer_rate)
consumed = random.randint(0, self.consumer_rate)
self.lag += produced - consumed
self.lag = max(0, self.lag)
self.history.append(self.lag)
return self.lag
def current_lag(self):
return self.lag
def average_lag(self):
if not self.history:
return 0
return sum(self.history) / len(self.history)
def is_healthy(self, threshold=10):
return self.lag <= threshold
if __name__ == "__main__":
gauge = ConsumerLagGauge(producer_rate=10, consumer_rate=7, initial_lag=5)
print("Timestamp | Lag | Healthy")
for i in range(6):
lag = gauge.tick()
print(f"{i:9} | {lag:3} | {gauge.is_healthy()}")
print(f"\nFinal lag: {gauge.current_lag()}")
print(f"Average lag: {gauge.average_lag():.1f}")
Output
Timestamp | Lag | Healthy
0 | 5 | True
1 | 5 | True
2 | 5 | True
3 | 5 | True
4 | 5 | True
5 | 5 | True
Final lag: 5
Average lag: 5.0
How it works
The ConsumerLagGauge class models a Kafka consumer that produces and consumes messages at random rates each tick. It tracks the cumulative lag as the difference between produced and consumed messages, clamped to zero to reflect that lag cannot be negative. History is stored in a deque with a maxlen of 10, so the average is computed over the most recent ticks. The is_healthy method provides a simple threshold check, mimicking real-world alerting rules. Random variation makes the output non-deterministic, so the exact printed numbers vary each run.
Common mistakes
- Forgetting to clamp lag to zero — negative lag is unrealistic
- Using a list instead of deque for history, losing memory bounds
- Computing average over all history instead of a sliding window
- Confusing 'lag' with total processed messages
Variations
- Use numpy to track time series and compute percentiles
- Add a logging or metrics emitter (e.g., Prometheus) to push lag
Real-world use cases
- Monitoring Kafka consumer lag in production to detect stuck or slow consumers.\n
- Testing alerting thresholds in a CI/CD pipeline before deploying monitoring changes.\n
- Simulating load scenarios to estimate acceptable consumer capacity for a new topic.\n
Sponsored
More from Observability & SRE
- Adding a Correlation ID to Log Context in Python medium
- Calculate Error Rate from Log Stream in Python easy
- Check if a Timestamp Falls in a Daily Maintenance Window in Python easy
- Export Metrics with OTLP Mock in Python medium
- Generate Mock CPU and Memory Metrics in Python easy
- Generate Prometheus Text Exposition Format in Python easy
Keep learning
Related tutorials and quizzes for this topic.