Alert on Key Metrics with Alertmanager

You've got dashboards full of metrics, but you only truly care when something breaks. Staring at graphs is not a monitoring strategy — it's the recipe for missed outages and 2 a.m. incident response chaos. This lesson shows you how to build a practical, rule-based alerting pipeline with Alertmanager, the component that turns raw metrics like node_cpu_seconds_total into actionable Slack, email, or webhook notifications. By the end, you'll confidently answer the question: "Is my system healthy right now?" without a single dashboard tab open.

The problem this lesson solves

Your server has a CPU spike. Your disk is filling up. Your API latency is skyrocketing. If you rely on a human to notice these signals on a graph, you've already lost. Dashboards are reactive — they require someone to watch them constantly, which doesn't scale beyond a single busy afternoon.

Alerting is the shift from watching to knowing. Instead of "I hope someone sees it," you get a proactive notification the moment a critical threshold is crossed. But alerting on raw metrics is noisy: a single 5-second CPU spike triggers ten emails, and you quickly learn to ignore every alert.

Alertmanager exists to solve this exactly. It's the decision engine that sits between your metrics and your notification channels. It deduplicates, routes, and suppresses alerts so you only get woken up when the cluster is genuinely on fire — not when one node sneezes.

Core concept / mental model

Think of your monitoring stack as a three-stage assembly line:

1. Collect — node_exporter or your application scrapes system metrics (CPU, memory, disk, HTTP latency) and exposes them over /metrics.
2. Evaluate — Prometheus runs alerting rules (defined in .rules.yml files) against those metrics, evaluating queries like up == 0 or node_filesystem_avail_bytes < 10% every 15 seconds.
3. Notify — When a rule fires, Prometheus pushes an alert to Alertmanager. Alertmanager applies routing, grouping, and deduplication, then sends a notification to your chosen receiver: Slack, email, PagerDuty, or a generic webhook.

Key distinction: Prometheus evaluates rules; Alertmanager handles notifications. You don't configure the what in Alertmanager — you configure the who, when, and how.

In Alertmanager, you define three core concepts:

• Receivers — the destination: email, slack, webhook, pagerduty. Each receiver has connection details, like SMTP server or Slack webhook URL.
• Routing trees — a decision tree that maps incoming alerts to receivers based on labels like severity, team, or environment.
• Grouping — a strategy to batch multiple related alerts into one notification, reducing noise. For example, group by alertname for Watchdog alerts.

Think of Alertmanager as the bouncer of your alerting party: it lets the important ones through, filters out the noise, and makes sure the right people get in.

How it works step by step

The flow from a metric to a notification involves these steps:

1. Define an alerting rule in Prometheus. The rule file is YAML with groups, rules, and an expr (PromQL expression) that evaluates to true when the condition is met.
2. Prometheus evaluates the rule at each scrape interval (default 15s). When the expression returns a value, the alert goes into pending state.
3. After a configurable for duration (e.g., 2 minutes), the alert transitions to firing — this prevents flapping.
4. Prometheus sends the firing alert to Alertmanager via an HTTP POST to /api/v1/alerts (Alertmanager URL is configured in prometheus.yml).
5. Alertmanager loads its routing tree and assigns the alert to the first matching route.
6. Alertmanager groups alerts with the same group key and deduplicates repeated notifications.
7. Alertmanager sends a notification to the configured receiver (Slack, email, etc.).
8. When the metric returns to normal, the rule resolves, and Alertmanager sends a resolved notification.

Hands-on walkthrough

Let's build a minimal-but-real alerting pipeline with Alertmanager. You'll need Docker or a Linux host with Prometheus and Alertmanager installed. We'll simulate a crash and watch the notification flow.

Step 1: Create an alerting rule

Create alerts.yml with a rule that fires when a node has been down for more than 1 minute:

# alerts.yml
groups:
  - name: instance_alerts
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} down"

Step 2: Configure Prometheus to use the rule and point to Alertmanager

Add the rule file and Alertmanager URL to prometheus.yml:

# prometheus.yml
global:
  scrape_interval: 15s

rule_files:
  - "alerts.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

Step 3: Configure Alertmanager with a receiver

Create alertmanager.yml with an email receiver and a simple route:

# alertmanager.yml
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'email'  # default
  routes:
    - matchers:
        - severity = critical
      receiver: 'email'
        # separate receiver for critical alerts
receivers:
  - name: 'email'
    email_configs:
      - to: 'oncall@example.com'
        from: 'alert@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alert@example.com'
        auth_password: 'secret'

Step 4: Run the stack

If you're using Docker, start all three services:

# docker-compose.yml or docker run commands
docker run -d --name prometheus -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml -v $(pwd)/alerts.yml:/etc/prometheus/alerts.yml -p 9090:9090 prom/prometheus
docker run -d --name alertmanager -v $(pwd)/alertmanager.yml:/etc/alertmanager/alertmanager.yml -p 9093:9093 prom/alertmanager
docker run -d --name node-exporter -p 9100:9100 prom/node-exporter

Step 5: Trigger an alert and observe the flow

Stop the node-exporter container to simulate a failure:

docker stop node-exporter

After a minute, check the Alertmanager UI at http://localhost:9093 — you should see the InstanceDown alert in firing state. If email is configured, you'll get a notification. If no SMTP, use a webhook receiver to see the JSON payload.

Pro tip: Use amtool (Alertmanager's CLI) to test routing without firing real alerts: amtool check-config alertmanager.yml validates your config.

Compare options / when to choose what

Alertmanager is the default choice in the Prometheus ecosystem, but there are alternatives. Here's a comparison:

When to choose what:

• Choose Alertmanager if you're already using Prometheus for scraping — it's the lowest-friction path and the natural place to manage alerts.
• Choose Grafana Alerting if you have multiple data sources (Loki logs, CloudWatch metrics) and want a single alerting surface in Grafana.
• Choose Kapacitor only if you're deeply invested in the InfluxDB stack and need anomaly detection beyond simple thresholds.

Troubleshooting & edge cases

• Alert fires but no notification — Check Alertmanager logs (docker logs alertmanager). Most common causes: wrong receiver name in route, firewall blocking port 9093 from Prometheus, or routing matcher not matching any labels. Validate with amtool check-config.
• Alert never reaches firing state — Your for duration is too long, or the expression has occasional gaps. Use amtool alert query to debug rule state. Try lowering for to 5s for testing.
• Flapping alerts — Frequent notifications. Increase repeat_interval (e.g., 4h) and group_wait (e.g., 30s) to batch alerts and avoid re-notifying on every evaluation cycle.
• Duplicate notifications across many alerts — Incorrect group_by settings. Use group_by: ['alertname', 'instance'] to group by alert and instance, so multiple instances down produce one notification, not many.
• Silenced but still firing — You may have silenced the wrong route. Check that the silence matchers match all labels of the alert (e.g., include instance=~"node.*"). Use amtool silence query to verify.
• Prometheus fails to start — Invalid rule file. Run promtool check rules alerts.yml to validate syntax.
• Alertmanager fails to load config — Validate with amtool check-config; common issues are missing route or receivers blocks.

What you learned & what's next

You now understand the complete alerting pipeline: Prometheus evaluates alerting rules, Alertmanager routes and notifies. You can define alert rules on key metrics, configure receivers (email, Slack, webhook), and troubleshoot common issues like flapping, misconfiguration, and notification failures.

You've also gained practical skills: writing PromQL expressions in alert rules, structuring alertmanager.yml routing trees, and using amtool/promtool for validation. These are the exact skills you'll build on next.

Next lesson: In the following tutorial, we'll explore alert routing and silences — mastering advanced grouping, dynamic label matching, and time-based silences to keep your on-call team sane during maintenance windows.