Set up Grafana dashboards

You’ve deployed pods, configured services, and maybe even wired up a Prometheus stack. But looking at raw PromQL queries in a terminal is like managing a live production system by reading its source code—possible, but not practical. Without a visual dashboard, you’re blind to trends, spikes, and slowdowns. This is where Grafana shines: it turns metric soup into clear, shareable, real-time charts that let you see your cluster’s health at a glance.

In this lesson, you’ll learn how to set up Grafana dashboards for Kubernetes metrics—connecting Grafana to your Prometheus data source, building your first dashboard panel, and importing battle-tested community dashboards.

The problem this lesson solves

Your Kubernetes cluster emits thousands of metrics every second: CPU usage, memory pressure, network traffic, pod restarts. Without a dashboard, you’d have to: - SSH into nodes and run kubectl top manually. - Query Prometheus with curl or kubectl exec every time you need a reading. - Cram data into spreadsheets if you want historical trends.

This approach doesn’t scale. By the time you notice a memory leak, it’s already degrading performance for users. Grafana dashboards solve this by giving you a single pane of glass—live, historical, and configurable alerts for every metric your team cares about.

Core concept / mental model

Think of Grafana as the front-end for your monitoring data, and Prometheus as the database.

• Prometheus collects and stores metrics (time-series data).
• Grafana connects to Prometheus and lets you build interactive charts, tables, and alert rules.

In Kubernetes terms, you usually deploy both as a stack: Prometheus scrapes metrics from pods/nodes, and Grafana reads from Prometheus. The architecture looks like this:

Kubernetes Cluster
├── Your App (emits metrics on /metrics)
├── Prometheus (scrapes and stores)
└── Grafana (queries Prometheus via data source)

Key terms

• Data source: A connection to a Prometheus (or other time-series) backend. Grafana supports Prometheus, InfluxDB, Graphite, and more.
• Dashboard: A collection of panels (charts, tables, etc.) arranged on a grid.
• Panel: A single visualization (e.g., line chart, gauge, stat).
• Query: The PromQL expression used to pull data from Prometheus into a panel.

How it works step by step

1. Deploy Grafana into your Kubernetes cluster (or use Grafana Cloud).
2. Add a Prometheus data source—point Grafana to your running Prometheus service.
3. Create or import a dashboard—either build panels from scratch or use community templates.
4. Configure queries using PromQL to select the right metrics (e.g., container_cpu_usage_seconds_total).
5. Visualize and share—set time ranges, refresh intervals, and optional alert thresholds.

Pro tip: Always start with one of the official Kubernetes dashboards to avoid reinventing the wheel. You can then clone and customize.

Hands-on walkthrough

Let’s set up Grafana on a minikube or real cluster. For this example, assume you already have Prometheus running in the monitoring namespace.

Step 1: Deploy Grafana

Create a deployment and service for Grafana. Use the official Docker image.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - containerPort: 3000
        env:
        - name: GF_SECURITY_ADMIN_PASSWORD
          value: "admin"
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  type: NodePort
  ports:
  - port: 3000
    targetPort: 3000
    nodePort: 30300
  selector:
    app: grafana

Apply:

kubectl apply -f grafana-deployment.yaml

Step 2: Access Grafana

Get your node IP and port:

kubectl get nodes -o wide   # get IP
# Open browser: http://<node-ip>:30300

Log in with admin / admin. Grafana will prompt you to change the password (you can skip for testing).

Step 3: Add Prometheus data source

1. In the left sidebar, click Configuration > Data Sources.
2. Click Add data source.
3. Choose Prometheus.
4. Set URL to your Prometheus service endpoint (e.g., http://prometheus.monitoring.svc.cluster.local:9090).
5. Click Save & Test. You should see a green success message.

Step 4: Import a Kubernetes dashboard

1. In the left sidebar, click + > Import.
2. Enter dashboard ID 315 (for “Kubernetes Cluster Monitoring (via Prometheus)”).
3. Select your Prometheus data source from the dropdown.
4. Click Import.

You now have a full cluster dashboard showing CPU, memory, network, and pod status.

Step 5: Build your first custom panel

1. Go to Dashboards > Manage and click New Dashboard.
2. Click Add new panel.
3. In the Query field, enter:

sum(rate(container_cpu_usage_seconds_total[5m])) by (pod)

1. Set a title like “CPU Usage Per Pod”.
2. Choose a line chart visualization.
3. Click Apply.

Your first custom dashboard panel is done!

Compare options / when to choose what

For most teams starting out, the recommendation is: import a community dashboard (e.g., ID 315) and then clone it to customize specific panels. This gives you a jumpstart without building from scratch.

Troubleshooting & edge cases

“No data” in panels

• Cause: Data source not reachable or metric doesn’t exist.
• Fix: Verify the Prometheus URL is correct. In the Explore tab of Prometheus, run container_cpu_usage_seconds_total to see if data exists. Also check that your Prometheus service name is resolvable inside the cluster.

Dashboard shows “NaN” or zero values

• Cause: Missing rate or duration in the query.
• Fix: For counter metrics (like _total), always apply rate() or increase(). Example: rate(container_cpu_usage_seconds_total[5m]).

Slow dashboard loading

• Cause: Too many panels querying heavy PromQL expressions.
• Fix: Reduce time range, use recording rules in Prometheus, or aggregate metrics before querying.

Can’t find the import dashboard ID

• Cause: The dashboards repository changed or ID deprecated.
• Fix: Visit grafana.com/grafana/dashboards and search for “Kubernetes”. Use the new ID from the website.

Pro tip: Always test a new dashboard on a non-production cluster first—some community dashboards are designed for large clusters and can overload small ones.

What you learned & what's next

You now know how to: - Deploy Grafana to your Kubernetes cluster. - Connect it to Prometheus as a data source. - Import a standard Kubernetes dashboard. - Build a simple custom chart using PromQL.

These skills give you real-time visibility into cluster health and let you surface key metrics (CPU, memory, throttling) without digging through command outputs.

What’s next? In the following lesson, you’ll learn how to configure alerting rules in Grafana and Prometheus—so you get notified when a pod enters CrashLoopBackOff or when memory exceeds 90%. Dashboard visibility combined with proactive alerts is what makes a monitoring system production-ready.