Use Prometheus for Model Metrics

Your model is in production, serving predictions, and everything looks fine — until a silent drift starts. Input distributions shift, latency spikes, or error rates climb. By the time stakeholders notice, the damage is done. You need observability, not just logs. This lesson shows you how to use Prometheus for model metrics, turning your ML service into a self-monitoring system that alerts you the moment something goes wrong — before your users feel it.

The problem this lesson solves

Machine learning models are not static. They degrade, they get hammered by new traffic patterns, and they fail in ways that standard application metrics miss. Health checks tell you if the process is alive, but not if the model is performing well. You can't catch data drift, prediction skew, or a silent drop in confidence with a simple /health endpoint.

The pain: Without model-specific metrics, you are flying blind in production. You discover issues from user complaints or, worse, from financial impact.

Logs are part of the answer, but they are passive. You have to go looking for problems. What you really need is a real-time, queryable, visualizable stream of model health data. That's where Prometheus fits. It gives you a time-series database, a powerful query language, and alerting — purpose-built for monitoring systems like your ML service.

Core concept / mental model

Think of Prometheus as a recording studio for your model. Your model is the musician, and each metric is a track. You get to choose which tracks to record: prediction counts, latency, feature distributions, confidence scores. The studio constantly records, and you can play back any moment in time — or set up alarms when a track goes out of tune.

Here's the key architecture:

• Instrumentation: Your Python code uses the prometheus_client library to define and update metrics (counters, gauges, histograms).
• Exposition: Your app exposes a /metrics HTTP endpoint that Prometheus scrapes on a schedule.
• Scraping: Prometheus server pulls metrics from your endpoint every N seconds (default 15s).
• Storage & Query: Prometheus stores the time-series data and lets you query it with PromQL (e.g., rate(model_predictions_total[5m])).
• Visualization & Alerting: Connect Grafana for dashboards, or use Alertmanager to fire alerts when thresholds break.

The pull-based model is a key difference from other monitoring tools. Prometheus asks your service for metrics, rather than your service pushing them somewhere. This makes discovery easy and scales well in dynamic environments like Kubernetes.

How it works step by step

Let's walk through the logic of wiring a model to Prometheus.

1. Define your metrics: Decide what matters. For most models, you'll want: - model_predictions_total — a counter of total requests - model_latency_seconds — a histogram of request latency - model_confidence — a gauge holding the latest average confidence score - model_feature_count — a gauge to catch feature-engineering bugs

2. Instrument the prediction function: Increment counters and observe histogram timings inside your /predict endpoint.

3. Expose the metrics endpoint: Create a Flask/FastAPI route that returns the text format Prometheus understands. The prometheus_client handles the serialization for you.

4. Configure Prometheus to scrape: Add a scrape_configs entry to prometheus.yml with your service's address and port.

5. Run Prometheus: Start the server, and it will begin collecting metrics on a loop.

6. Query with PromQL: Write expressions like rate(model_predictions_total[5m]) to see request volume, or histogram_quantile(0.95, sum(rate(model_latency_seconds_bucket[5m])) by (le)) for p95 latency.

Hands-on walkthrough

Let's build a small FastAPI service that serves a fake model — but the pattern is identical for any real model (scikit-learn, PyTorch, TensorFlow). We'll expose three metrics and prove Prometheus can scrape them.

Step 1: Install dependencies

pip install fastapi uvicorn prometheus-client

Step 2: The application code

# app.py
from fastapi import FastAPI
from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST
from prometheus_client import start_http_server
from fastapi.responses import Response
import time
import random

app = FastAPI()

# Define metrics
PREDICTIONS = Counter('model_predictions_total', 'Total number of predictions made')
LATENCY = Histogram('model_latency_seconds', 'Latency of prediction requests')
CONFIDENCE = Gauge('model_confidence', 'Latest average confidence score')

@app.get('/metrics')
def metrics():
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)

@app.get('/predict')
def predict():
    start = time.time()

    # Fake inference — replace with real model.predict()
    time.sleep(random.uniform(0.01, 0.1))
    confidence = random.uniform(0.8, 0.99)

    # Record metrics
    PREDICTIONS.inc()
    LATENCY.observe(time.time() - start)
    CONFIDENCE.set(confidence)

    return {'prediction': random.randint(0, 1), 'confidence': confidence}

if __name__ == '__main__':
    import uvicorn
    uvicorn.run(app, host='0.0.0.0', port=8000)

Step 3: Run the service

uvicorn app:app --host 0.0.0.0 --port 8000

Make a few requests to /predict, then curl the metrics endpoint:

for i in {1..100}; do curl -s localhost:8000/predict > /dev/null; done
curl localhost:8000/metrics | grep -E "^(model_predictions_total|model_latency_seconds_bucket|model_confidence)"

You'll see output like:

# HELP model_predictions_total Total number of predictions made
# TYPE model_predictions_total counter
model_predictions_total 100
# HELP model_latency_seconds Latency of prediction requests
# TYPE model_latency_seconds histogram
model_latency_seconds_bucket{le="0.005"} 0
model_latency_seconds_bucket{le="0.01"} 0
model_latency_seconds_bucket{le="0.025"} 3
...
model_latency_seconds_bucket{le="+Inf"} 100
# HELP model_confidence Latest average confidence score
# TYPE model_confidence gauge
model_confidence 0.922

Step 4: Configure Prometheus

Create prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'ml-model'
    static_configs:
      - targets: ['localhost:8000']

Run Prometheus (download binary or use Docker):

docker run -p 9090:9090 -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Open http://localhost:9090, and in the query box try:

rate(model_predictions_total[1m])

You'll see the request rate over the last minute. Try histogram_quantile(0.95, sum(rate(model_latency_seconds_bucket[5m])) by (le)) for the 95th percentile latency.

Compare options / when to choose what

When to choose Prometheus: If you need open-source, self-hosted monitoring with powerful querying and alerting, or if you're already in Kubernetes (Prometheus is the de facto standard).

When to choose something else: For experiment tracking, use MLflow. For simple, low-volume apps, StatsD might be enough. For deep AWS integration, CloudWatch simplifies operations.

Troubleshooting & edge cases

• Metrics endpoint returns 404: Forget to add the /metrics route to your app? If you're using start_http_server(8001) separately, you might expose on a different port. Keep one endpoint.

• Prometheus can't scrape — connection refused: Check your service's port and whether it's binding to 0.0.0.0 (not just 127.0.0.1). In a container, ensure the port is published.

• Counter reset on restart: Counters are process-local. If your app restarts, the counter starts from zero. Prometheus still handles this via rate(), but you'll lose absolute totals. Consider using a gauge of a database counter if you need persistence.

• High cardinality explosion: Adding labels that vary wildly (e.g., user ID) creates a metric family per label value, which can blow up memory. Keep labels to model version, environment, or endpoint.

• Latency histogram bucket misconfiguration: If your buckets don't cover your latency range, you'll lose accuracy in percentiles. Define buckets that span your expected distribution, e.g., Histogram('latency', '...', buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1)).

• Scrape interval too slow: Default 15s might miss short spikes. For critical models, reduce to 5s, but be mindful of load.

Pro tip: Use prometheus_client's start_http_server to expose metrics on a separate port if you want to keep your service port clean.

What you learned & what's next

You now know how to use Prometheus for model metrics. You can define counters, histograms, and gauges; expose them via a /metrics endpoint; and query them with PromQL to understand your model's health in production. This is your first step toward a full observability stack.

In the next lesson, you'll build on this foundation by adding alerting with Alertmanager — turning your metrics into automatic pager notifications when thresholds are breached. You'll also learn to integrate Grafana for rich dashboards, so your entire team can see model performance at a glance.

Everything you've learned here — choosing meaningful metrics, understanding the pull model, and writing PromQL — will be directly reusable. You're no longer flying blind. Your model now has a voice, and you're listening.