Create custom metric alerts

You've built your Azure environment, deployed apps, and wired up networking — but when something breaks, how do you know before your users do? Watching dashboards manually is a losing game. In this lesson, you'll learn to create custom metric alerts that watch your key performance indicators and page you the moment they drift out of bounds — so you can fix problems while they're still quiet, not after the incident page explodes.

The problem this lesson solves

Without alerts, you're flying blind. Your app could be slowly leaking memory, your database could be approaching its connection limit, or your API could be returning 500s to a small but growing set of users. By the time anyone notices, the issue has often escalated into a full-blown incident.

Custom metric alerts give you a programmatic tripwire. You define a condition — say, CPU usage greater than 80% for five minutes — and Azure monitors it continuously, firing a notification when the condition is met. This isn't just about avoiding downtime; it's about operational maturity. Teams that respond to alerts before customers complain are happier, more productive, and less stressed.

Alerting also matters for cost control. A sudden spike in outbound data transfer might be a sign of an inefficient query or a runaway process. Alerting on those metrics helps you catch cost leaks early.

The core problem this lesson solves is simple: how to turn raw telemetry into actionable signals that trigger the right people at the right time.

Core concept / mental model

Think of Azure Monitor as the central nervous system of your Azure resources. Every service emits metrics — numerical values like CPU percentage, request count, or queue length. These metrics are stored for a rolling period (typically 30 days for most services).

A metric alert is a rule that watches one of these metrics and compares it to a threshold. Think of it like a thermostat: you set a desired range, and when the temperature (your metric) crosses the threshold, the furnace or AC (your alert action) kicks in.

Key terms you'll encounter: - Metric — a single measurable value emitted by a resource. - Metric alert rule — a definition that includes the metric, condition, window, and action group. - Condition — the logical operation, like "greater than" or "less than", applied to the metric value over a time window. - Action group — a set of actions (email, SMS, webhook, ITSM) that trigger when the alert fires. - Alert state — Fired, Resolved, or Acknowledged.

The anatomy of a metric alert

A metric alert works in three phases: 1. Collect — Azure Monitor ingests metrics from your resource. 2. Evaluate — At a frequency you set (e.g., every 1 minute), the alert engine checks if the condition is true. 3. Act — If the condition is true, the alert is fired and the action group is invoked.

This model is consistent across resource types, which is why once you master one alert, you can quickly build others.

How it works step by step

Creating a custom metric alert involves a series of logical steps. Let's break them down.

Step 1: Identify the metric you care about

First, decide which metric best represents the health of your resource. For an App Service, it might be CpuTime or Requests. For a Storage account, Ingress or Egress. For a Virtual Machine, Percentage CPU or Network In Total.

You can see available metrics in the Azure Portal by navigating to your resource → Monitoring → Metrics. Scroll through the list and note the name (and unit) of the metric you want.

Step 2: Define the threshold and window

Ask: What value would indicate a problem? This depends on your workload. A burst of CPU to 90% for one minute might be normal for a batch job; an average above 80% for 30 minutes is likely problematic.

You'll set a threshold (the value) and an evaluation window (how long the condition must hold before firing). For example:

• Greater than 80% CPU for 10 minutes
• Less than 2 requests per minute for 5 minutes

Step 3: Choose the aggregation

Metrics are raw samples. You can aggregate them over the window using Average, Minimum, Maximum, Total, or Count. For steady-state resources, Average is typical. For detecting latency spikes, Maximum is better.

Step 4: Set an evaluation frequency

How often should Azure check the condition? Every 1 minute is granular, but it costs a bit more. Every 5 minutes is often sufficient. Balance alert responsiveness with cost.

Step 5: Configure the action group

An alert that no one sees is worthless. Create an Action group with your preferred channels — email, SMS, webhook, ITSM. You can reuse the same action group across many alerts.

Step 6: Create the alert rule

Finally, assemble the rule via Portal, CLI, or ARM template. Each method is valid — choose the one that fits your workflow.

Hands-on walkthrough

Let's create a custom metric alert on an Azure Virtual Machine, watching Percentage CPU. We'll use the Azure CLI because it's scriptable and repeatable.

Prerequisites

• An Azure subscription
• An existing VM named myVM (or any resource)
• Azure CLI installed and logged in

Example 1: Create a metric alert via CLI

First, create an action group that sends email to ops@example.com:

az monitor action-group create \
  --resource-group myResourceGroup \
  --name myActionGroup \
  --short-name myAlertGroup \
  --action email ops-email ops@example.com

Then create the alert rule:

az monitor metrics alert create \
  --name "High CPU" \
  --resource-group myResourceGroup \
  --scopes /subscriptions/<sub-id>/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachines/myVM \
  --condition "Percentage CPU > 80 avg 5m" \
  --action-groups /subscriptions/<sub-id>/resourceGroups/myResourceGroup/providers/microsoft.insights/actionGroups/myActionGroup \
  --evaluation-frequency 1m \
  --window-size 5m \
  --auto-mitigate true

Expected output (condensed):

{
  "name": "High CPU",
  "condition": {
    "metricName": "Percentage CPU",
    "operator": "GreaterThan",
    "threshold": 80,
    "timeAggregation": "Average",
    "windowSize": "PT5M"
  },
  "enabled": true
}

The rule is now active. If CPU averages above 80% over any 5-minute window, the action group will fire an email.

Example 2: Create an alert with a dynamic threshold

Dynamic thresholds use machine learning to detect anomalies instead of a fixed number. This is great for metrics with seasonal patterns.

az monitor metrics alert create \
  --name "Dynamic CPU Anomaly" \
  --resource-group myResourceGroup \
  --scopes /subscriptions/<sub-id>/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachines/myVM \
  --condition "Percentage CPU dynamic 5m" \
  --action-groups /subscriptions/<sub-id>/resourceGroups/myResourceGroup/providers/microsoft.insights/actionGroups/myActionGroup

This rule has no fixed threshold; it learns the normal baseline and alerts on deviations.

Example 3: Deploy alert rules as infrastructure with Bicep

For production, define alerts in code so they're versioned and repeatable. Here's a Bicep snippet:

resource alertRule 'Microsoft.Insights/metricAlerts@2018-03-01' = {
  name: 'High-CPU-Alert'
  location: 'global'
  properties: {
    description: 'Alert when CPU is high'
    severity: 3
    enabled: true
    scopes: [
      vmId
    ]
    evaluationFrequency: 'PT1M'
    windowSize: 'PT5M'
    criteria: {
      'odata.type': 'Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria'
      allOf: [
        {
          name: 'Metric1'
          metricName: 'Percentage CPU'
          operator: 'GreaterThan'
          threshold: 80
          timeAggregation: 'Average'
        }
      ]
    }
    actions: [
      {
        actionGroupId: actionGroupId
      }
    ]
  }
}

Deploy with:

az deployment group create --resource-group myResourceGroup --template-file main.bicep

This IaC approach makes your alerts reviewable, auditable, and deployable to multiple environments.

Compare options / when to choose what

You have several ways to create metric alerts. Here's a quick comparison:

Our recommendation: Start in the Portal to understand the concepts, then move to CLI for rapid iteration, and finally commit to Bicep for production. The investment in IaC pays off when you need to recreate alerts across environments or trace changes.

Variations on alert conditions

• Static threshold — fixed value; simplest and most predictable.
• Dynamic threshold — machine-learning based; adapts to seasonality but may require tuning.
• Multi-condition — combine several metrics (e.g., CPU > 80% AND memory > 90%); more complex but more precise.

Troubleshooting & edge cases

Alert not firing

• Check that the resource actually emits metrics. Some metrics appear only when activity occurs (e.g., a VM that's stopped has no CPU data).
• Verify the threshold is realistic. If you set CPU > 90% but your app runs at 30%, it will never fire.
• Confirm the action group is valid and has active channels. Email can land in spam.

Alert fires too often

• "Flapping" happens when your metric lingers right near the threshold. Increase the window size (e.g., from 5 to 15 minutes) or use dynamic thresholds.
• Check the aggregation — Average is smoother than Maximum. Using Maximum over a 1-minute window may catch transient spikes.

Legal/security issues

• For compliance, ensure alerts meet your retention and privacy policies. Alert emails may contain resource names — keep them non-sensitive.

Wrong metric

• The metric name is case-sensitive in API calls. In Portal, metrics often have friendly names like "CPU Utilization" but the underlying metric name is Percentage CPU. Always verify with the CLI or metrics API.

Action group not triggering

• Test the action group independently — use Test action group in Portal to send a test email.

What you learned & what's next

You now know how to create custom metric alerts — the art of turning raw telemetry into actionable notifications. You've learned the mental model (metric adductors → alert rule → action group), the step-by-step process, and how to implement it with CLI, Portal, and Bicep. You can troubleshoot the most common traps and choose static vs dynamic thresholds.

This is a foundational skill for any Azure operator. In the next lesson, you'll build on this by diving into Azure Log Analytics queries — combining metric alerts with log-based alerts for even richer monitoring. You'll be able to alert not just on numbers but on log messages, giving you visibility into application errors and exceptions. Keep your alerting sharp and your on-call schedule quiet.

Key objectives met: - You can explain the core idea behind custom metric alerts. - You completed a practical exercise (CLI + Bicep). - You know how to connect this knowledge to the next step in your Azure learning path.