Grafana Dashboards

You've built the AI model, wired up the API, and stored the results in a database — but when your manager asks, "What's happening with the system?", you freeze. You can't show any metrics, you can't explain the latency spike, and you definitely can't prove that your new prompt optimization actually improved output quality. The pain is real: a system you can't observe is a system you can't trust. In this lesson you'll learn how to build dashboards with Grafana — the open-source observability platform that turns your raw metrics into beautiful, live, shareable dashboards. You'll go from zero to a working dashboard that tracks the health and performance of an AI-powered application, using Python to generate the data and Grafana to visualize it.

A Grafana dashboard is like the cockpit of an airplane. Instead of gauges and dials showing speed and altitude, you're looking at panels — each panel is a chart, table, or alert that displays data from a data source. The data source is where your metrics live, such as Prometheus, InfluxDB, or even a simple SQL database. Grafana doesn't store your data itself; it's a presentation layer that queries the source on demand. This separation is key: your application just sends metrics to a time-series database, and Grafana takes care of the rest. The mental model is simple:

• Data source: where your metrics live (e.g., Prometheus).
• Query: Grafana asks the data source for a subset of metrics over a time window.
• Panel: a visualization of that query result (graph, table, stat).
• Dashboard: a collection of panels arranged on a grid, with shared time controls.

For AI engineers, this means you can monitor model latency, error rates, prediction distributions, and resource utilization — all in one place. You can even set up alert rules to be paged when accuracy drops below a threshold. Grafana is the reason your AI system goes from a black box to a glass box.

Now let's see the actual flow of building a dashboard, step by step. First, you need a running Grafana instance. The easiest way is via Docker:

docker run -d -p 3000:3000 --name grafana grafana/grafana:latest

Next, you need a data source. For this lesson we'll use Prometheus, a time-series database perfect for metrics. Run it with:

docker run -d -p 9090:9090 --name prometheus prom/prometheus:latest

Before starting Grafana, make sure Prometheus has a target to scrape. We'll create a simple Python app that exposes metrics via the prometheus_client library. Here's a complete example:

# metrics_app.py
from prometheus_client import start_http_server, Summary, Counter, Gauge
import random
import time

REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')
REQUESTS = Counter('http_requests_total', 'Total HTTP requests')
ERRORS = Counter('http_errors_total', 'Total HTTP errors')
IN_PROGRESS = Gauge('http_requests_in_progress', 'In progress requests')

start_http_server(8000)  # Exposes metrics at http://localhost:8000/metrics

while True:
    REQUESTS.inc()
    IN_PROGRESS.inc()
    time.sleep(random.uniform(0.5, 1.5))
    with REQUEST_TIME.time():
        time.sleep(random.uniform(0.1, 0.5))
    # Simulate an error with ~10% probability
    if random.random() < 0.1:
        ERRORS.inc()
    IN_PROGRESS.dec()

Run this script in the background, then point Prometheus at it by creating a prometheus.yml file:

scrape_configs:
  - job_name: 'python-app'
    static_configs:
      - targets: ['host.docker.internal:8000']

The host.docker.internal works on Docker for Mac/Windows. On Linux, you'll need to use your host IP or run the app in another container on the same network. With that in place, you can now start Grafana and add Prometheus as a data source. The default Grafana login is admin/admin. Go to Configuration → Data Sources → Add data source → Prometheus, and enter http://localhost:9090.

Once connected, the fun begins: creating panels. Here's how to add a simple panel showing request latency:

• Click Dashboards → New dashboard → Add panel.
• In the Query editor, select your Prometheus data source and enter:
• rate(request_processing_seconds_sum[1m]) / rate(request_processing_seconds_count[1m])
• This formula calculates the average latency over the last minute. Choose Time series visualization, set a title like "Avg Request Latency", and press Apply.

You'll immediately see a live line chart. Repeat for error rate: rate(http_errors_total[1m]) / rate(http_requests_total[1m]). Add a Stat panel showing the latest value of http_requests_total. Your dashboard is alive — refresh the page and watch the metrics flow.

But Grafana isn't limited to Prometheus. Here's a quick comparison of data source options:

For most AI applications, Prometheus is the go-to because it's the standard for cloud-native monitoring. If you need to visualize something like user click logs, use PostgreSQL. For real-time sensor data, InfluxDB shines. The choice depends on your data type and query complexity.

Now, even with a working dashboard, things can go wrong. Here are the most common pitfalls:

• Metrics not appearing: Check that your /metrics endpoint is reachable from the Prometheus container. Use curl http://localhost:8000/metrics on the host — if it fails, look at your firewall or Docker network settings.
• Data source connection error: In Grafana, if the data source says "Error reading Prometheus: Post \"http://localhost:9090/api/v1/query\": dial tcp ... connection refused", it's likely because Prometheus is on a different network. Use the host's IP address or host.docker.internal. Ensure both containers are on the same network if you use container names.
• Empty panels: PromQL queries can return no data if the metric name is wrong. Double-check your metric names by visiting http://localhost:8000/metrics and looking for the exact name. For example, http_requests_total vs http_requests. Also remember that counters need rate() or increase() to be useful; showing a cumulative total will just be a constantly increasing line.
• Dashboard not refreshing: If you expect live updates but see a static chart, set the refresh interval in the dashboard settings (Settings → Time options → Auto refresh). Default is often 5s, but it can be set to off.

One edge case that trips up AI engineers: high cardinality. If you label your metrics with something like user_id or prompt_hash, Prometheus will create a separate time series for each unique label value. This can explode in memory and slow down queries. Stick to low-cardinality labels like model_name, version, and environment. Also, be careful with histogram buckets for latency — choose meaningful thresholds (e.g., 100ms, 250ms, 500ms) that match your service's SLA.

You've now mastered the core of building dashboards with Grafana. You understand the architecture: your Python app exposes metrics via the Prometheus client, Prometheus scrapes them, and Grafana visualizes them in panels. You've run a hands-on exercise that gave you a live latency chart and error rate. You've compared Prometheus, InfluxDB, and PostgreSQL for different use cases, and you know how to fix the most common issues. This is a skill that transforms you from a model-builder into a production engineer.

Now that you can build dashboards, the natural next step in the Applied AI engineering track is to learn how to set up alerts on top of your dashboards — so you get paged when ValueError rate spikes or GPU memory runs out. Alerting is where Grafana really earns its keep. But first, go build a dashboard for one of your existing projects. Export a JSON definition and share it with your team. You'll be amazed at how much clarity it brings.