Monitor Cluster with Prometheus

When your Kubernetes cluster goes down, do you rely on guesswork? Without observability, you're flying blind — unable to see CPU spikes, memory leaks, or failing pods until it's too late. The Prometheus stack (kube-prometheus-stack) is the gold standard for monitoring clusters with Prometheus, collecting metrics from nodes, pods, and services, and alerting you before users feel the pain.

The problem this lesson solves

Kubernetes is dynamic: pods die, nodes scale, and workloads shift. Traditional monitoring tools (like Nagios) expect static IPs and fixed services — they break in a cluster that auto-heals. Manual kubectl top commands give a snapshot, not history. You need a system that:

• Discovers targets automatically via Kubernetes APIs
• Stores time-series metrics (CPU, memory, network, disk)
• Alerts when thresholds cross (e.g., node disk pressure)
• Visualizes trends over time (via Grafana)

The Prometheus stack solves all of this in one cohesive deployment.

Core concept / mental model

Think of Prometheus as a polling camera, not a log file. Instead of waiting for errors, Prometheus periodically scrapes HTTP endpoints (/metrics) exposed by applications and the cluster itself. Each scrape pulls key-value pairs called metrics (e.g., container_memory_usage_bytes).

Metrics flow like this:

1. Exporters (on nodes, pods, or third-party services) expose /metrics.
2. Prometheus server scrapes those endpoints at a configurable interval (default 30s).
3. The data is stored in a time-series database and queried via PromQL.
4. Alertmanager fires notifications (email, Slack, PagerDuty) when PromQL rules trigger.
5. Grafana visualizes metrics with dashboards.

Pro tip: The default kube-prometheus-stack Helm chart bundles Prometheus, Alertmanager, and Grafana into one launch — you get a full observability stack out of the box.

Key terms to understand

• Scrape interval — how often Prometheus polls targets (adjust for cost vs granularity).
• PromQL — the query language for slicing metrics (e.g., sum(rate(container_cpu_usage_seconds_total[5m]))).
• Alert rules — PromQL expressions that evaluate to pending or firing alerts.
• ServiceMonitor — a Kubernetes CRD that tells Prometheus which services to monitor.

How it works step by step

1. Install the stack using Helm or raw manifests. Helm is recommended because it manages upgrades, ConfigMaps, and RBAC.

2. Configure ServiceMonitors to scrape cluster components (kube-controller-manager, etcd, kube-proxy) and your own applications.

3. Add custom exporters — e.g., Node Exporter for node metrics, kube-state-metrics for pod states.

4. Create alerting rules — Prometheus supplies default community rules for common issues (e.g., high memory, pod crash looping).

5. Set up Grafana dashboards — import pre-built dashboards (like Kubernetes Cluster Monitoring via 315) or build custom panels.

6. Define notification receivers in Alertmanager — Slack webhook, email SMTP, or OpsGenie.

The stack automatically discovers nodes and services via Kubernetes service discovery — no manual target lists.

Hands-on walkthrough

Install kube-prometheus-stack with Helm

First, add the repository and install the chart into a dedicated monitoring namespace:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create namespace monitoring

helm install monitor-cluster prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set grafana.enabled=true \
  --set grafana.service.type=LoadBalancer

Expected output (similar):

NAME: monitor-cluster
LAST DEPLOYED: ...
NAMESPACE: monitoring
STATUS: deployed
...

Verify the components are running

kubectl get pods -n monitoring

You should see pods like: - monitor-cluster-kube-state-metrics-* - monitor-cluster-prometheus-node-exporter-* - monitor-cluster-grafana-* - prometheus-monitor-cluster-kube-prome-prometheus-0

Access Grafana

Get the LoadBalancer IP:

kubectl get svc -n monitoring | grep grafana

Open http://<EXTERNAL-IP>:80 in your browser. Default credentials: - Username: admin - Password: prom-operator

Once logged in, explore the “Kubernetes / Cluster” dashboard. You’ll see real-time CPU, memory, disk I/O, and network traffic across all nodes.

Query a metric in Prometheus

Port-forward Prometheus server:

kubectl port-forward svc/monitor-cluster-kube-prome-prometheus 9090 -n monitoring

Open http://localhost:9090, and run a PromQL query:

sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace)

This returns CPU usage per namespace over the last 5 minutes.

Compare options / when to choose what

Pro tip: For small dev clusters, the standalone Prometheus Operator is lighter. For production, kube-prometheus-stack saves you hours of wiring.

Variations to consider

• Use kube-prometheus (the old deprecated project) vs the newer kube-prometheus-stack — always choose the stack.
• Deploy without Grafana and point to an existing Grafana instance.
• Enable persistent storage for Prometheus to survive pod restarts (set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate).

Troubleshooting & edge cases

Scraping fails: endpoints not found

Check that your apps expose a /metrics endpoint returning valid Prometheus format. Test with curl <pod-ip>:<port>/metrics from within the cluster. Also verify the ServiceMonitor label selector matches your service labels.

High memory usage from Prometheus

Prometheus is memory-intensive when scraping thousands of targets. Reduce the scrape interval to 60s and enable sample limit in your ServiceMonitor (.spec.endpoints[0].metricRelabelings).

Alertmanager not sending notifications

Check Alertmanager configuration. Ensure the receivers section is correct and webhook URLs are reachable from inside the cluster. Test with a simple alert like ALERT TestAlert IF 1 == 1 FOR 1m.

Common mistakes

• Not setting resource limits — the default heap size can cause OOMKill for large clusters.
• Using default passwords — change Grafana admin password immediately.
• Ignoring etcd metrics — etcd latency is critical for control plane health but often not scraped by default.
• Skipping node exporter — without it, you miss node-level disk I/O and network errors.

What you learned & what's next

You can now monitor your cluster with Prometheus stack: install it with Helm, access Grafana dashboards, query metrics with PromQL, and understand the roles of ServiceMonitors and exporters. You’ve seen how to troubleshoot basic scraping issues and compare deployment options.

Next up: Lesson 60 — Set up alerting rules — where you’ll write custom alerting expressions to get notified before failures happen.