Canary Deployments with Flagger

You know the sinking feeling. You push a new version to production, watch the dashboard for five minutes, and hope nothing breaks. If something does, you're either scrambling to revert a Deployment or explaining to the on-call engineer why the old service is still running on pod IPs the new one doesn't know about. Canary deployments eliminate that panic by letting you shift a tiny percentage of real traffic — 1%, 5%, 10% — to the new version while keeping the old one serving the rest. When the new version looks healthy, the shift continues; when it doesn't, Flagger rolls back for you. This lesson shows you how to wield that power with Flagger in Kubernetes.

The problem this lesson solves

A traditional Kubernetes Deployment update is all-or-nothing. You update the image tag, the controller creates new Replicas, and traffic immediately switches to them as soon as the new Pods are Ready. If the new code has a subtle bug — a slow memory leak, a misconfigured environment variable, a query that deadlocks under 1% of requests — you don't find out until users start complaining. By then, the old Pods are gone.

Canary deployments solve this by decoupling deployment from traffic. You deploy the new version alongside the old one, but the Kubernetes Service or Ingress sends only a small fraction of requests to the new Pods. You watch metrics (HTTP error rate, latency, CPU) for a few minutes. If nothing breaks, Flagger increases the traffic share. If the error rate spikes, Flagger sends all traffic back to the old version and marks the canary as failed. No manual intervention required.

Core concept / mental model

Think of a canary in a coal mine — a small, sensitive indicator of danger. In Kubernetes, Flagger is your canary. It watches the health of the new version and decides when to let more traffic through.

Flagger is a progressive delivery tool that works with service mesh (Istio, Linkerd, Consul Connect) or ingress controllers (NGINX, Contour, Gloo). It uses a custom resource called Canary to define the deployment strategy. The Canary object tells Flagger which Deployment to watch, what metrics to check, and how long to wait between traffic shifts.

The mental model is simple: 1. Deploy the old version (baseline). 2. Deploy the new version (primary). 3. Flagger creates a canary Deployment that runs the new image with no traffic. 4. Flagger gradually shifts traffic from baseline to canary, checking metrics after each step. 5. If any step fails the health check, the canary is aborted and traffic stays on the old version. 6. If all steps pass, Flagger promotes the canary — updates the target Deployment to the new image and deletes the canary Deployment.

This is progressive delivery, not just canary deployments. Flagger adds the automation and observability that makes canaries practical in production.

How it works step by step

1. Install Flagger and a metric provider

Flagger needs to read metrics from Prometheus or a similar service. The easiest path is to install Flagger alongside Prometheus and one of its supported meshes or ingresses. The Flagger Helm chart handles all of this.

2. Define the target Deployment

Your application Deployment must have a stable name. Flagger will use this Deployment as the baseline. The new version goes into a new Deployment that Flagger creates and manages.

3. Create a Canary custom resource

The Canary object is where you specify: - TargetRef: the name of the Deployment to canary - Service: the Kubernetes Service that should switch traffic - Analysis: the interval between steps, the maximum number of steps, and the metrics to check - ProgressDeadlineSeconds: how long the entire canary can take before Flagger gives up

Here's a minimal Canary YAML:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: my-app-canary
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  service:
    port: 80
    targetPort: 8080
    name: my-app-svc
  analysis:
    interval: "1m"
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: "1m"
      - name: request-duration
        thresholdRange:
          max: 500
        interval: "1m"
    webhooks:
      - name: "load-test"
        type: pre-rollout
        url: http://load-tester.test/start
        timeout: 30s

4. Flagger begins the canary

When you update the Deployment's image tag (or any other spec field), Flagger detects the change and: - Scales down the canary to 0 replicas - Scales up the canary Deployment with the new image - Adds the canary Pods to the Service - Shifts traffic in steps — 10%, 20% … up to maxWeight

5. Health checks and rollback

After each step, Flagger waits for the interval duration, then checks the metrics. If the success rate drops below 99% or the latency exceeds 500ms (per your thresholds), Flagger aborts the canary. It sets the canary's weight to 0, scales down the canary Deployment, and updates the primary Deployment to the old image.

6. Promotion

If all steps pass, Flagger promotes the canary — it updates the original Deployment's image tag to the new version and scales down the canary. The canary is complete.

Hands-on walkthrough

You need a Kubernetes cluster with Istio or NGINX Ingress installed, plus Flagger and Prometheus deployed. Here we'll use NGINX Ingress because it's simpler for demos.

Step 1: Install Flagger and dependencies

# Add the Flagger Helm repo
helm repo add flagger https://flagger.app

# Install Flagger with NGINX Ingress support
helm upgrade -i flagger flagger/flagger \
  --namespace=istio-system \
  --set crd.create=false \
  --set meshProvider=nginx \
  --set metricsServer=http://prometheus-kube-prometheus-prometheus.monitoring:9090

Step 2: Deploy a sample application

kubectl create ns test
kubectl apply -f https://raw.githubusercontent.com/fluxcd/flagger/main/artifacts/loadtester/deployment.yaml -n test
kubectl apply -f https://raw.githubusercontent.com/fluxcd/flagger/main/artifacts/loadtester/service.yaml -n test

# Deploy the primary app
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
  namespace: test
spec:
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      labels:
        app: podinfo
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:6.0.0
          ports:
            - containerPort: 9898
              name: http
          command:
            - ./podinfo
            - --port=9898
---
apiVersion: v1
kind: Service
metadata:
  name: podinfo-svc
  namespace: test
spec:
  selector:
    app: podinfo
  ports:
    - name: http
      port: 9898
      targetPort: 9898
EOF

Step 3: Create the Canary object

cat <<EOF | kubectl apply -f -
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo-canary
  namespace: test
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  service:
    port: 9898
    targetPort: 9898
    name: podinfo-svc
  ingressRef:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    name: podinfo-ingress
  analysis:
    interval: 30s
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 30s
      - name: request-duration
        thresholdRange:
          max: 500
        interval: 30s
    webhooks:
      - name: load-test
        type: pre-rollout
        url: http://flagger-loadtester.test/start
        timeout: 30s
EOF

Step 4: Trigger a canary

Update the image to a new version:

kubectl set image deployment/podinfo -n test \
  podinfo=ghcr.io/stefanprodan/podinfo:6.1.0

Step 5: Watch the canary progress

kubectl describe canary/podinfo-canary -n test
kubectl get canary/podinfo-canary -n test -w

Expected output (after 2 minutes, assuming all checks pass):

NAME              STATUS      WEIGHT   LASTTRANSITIONTIME
podinfo-canary   Initialized   0        2023-10-01T12:00:00Z
podinfo-canary   Progressing   10       2023-10-01T12:01:00Z
podinfo-canary   Progressing   20       2023-10-01T12:01:30Z
...
podinfo-canary   Promoting     50       2023-10-01T12:05:00Z
podinfo-canary   Succeeded     100      2023-10-01T12:05:30Z

If a metric fails, you'll see Failed status and the deployment will not progress.

Compare options / when to choose what

Choose Flagger when you want an operator that watches metrics automatically and doesn't require you to patch VirtualServices or change Service selectors by hand. Choose Argo Rollouts if you're already in the Argo ecosystem and want more control over the analysis (manual judgment, experiments).

Troubleshooting & edge cases

"Canary stuck at initializing"

Flagger hasn't created the canary Deployment yet. Check: - TargetRef points to a real Deployment that exists? - The Deployment's spec is complete (no missing ports, etc.)? - Flagger has permissions to create Deployments in that namespace?

"Failed to get metrics"

Flagger can't reach Prometheus. Verify: - The metricsServer URL in Flagger's Helm configuration is correct. - Prometheus is reachable from Flagger's Pod (try wget http://prometheus:9090/api/v1/query?query=up). - The metric names (like request-success-rate) exist in Prometheus. Flagger needs the metric to exist before the canary starts — if it's your first canary, the metric won't exist until traffic flows.

"Canary aborted — failure threshold exceeded"

Your app is failing the health check. Look at: - The metric thresholds in the analysis — are they too tight? A brief spike during deployment can trigger rollback. - The interval — too short? Give the new Pods time to warm up. - The webhook load tester — is it actually sending traffic? If you don't have a load tester, Flagger won't see real metrics.

Edge case: Zero traffic canary

If your app has no real users (staging environment), you can still simulate traffic with a simple shell loop:

while true; do
  curl -s http://podinfo-svc.test:9898/ | grep -c "version" || true
  sleep 1
done

What you learned & what's next

You now know how to implement canary deployments with Flagger — from installing Flagger, defining a Canary custom resource, triggering a canary, and watching the automated traffic shift and rollback. You can compare Flagger to Argo Rollouts and manual approaches. You can troubleshoot common issues like missing metrics, stuck initialization, and failed health checks.

Next step: Automate canaries further using GitOps with Flagger and Flux — where a Git commit triggers the canary instead of a kubectl set image command. That's the next lesson in this track.

Pro tip: Start with a simple canary in a non-production namespace. Use the stepWeight: 5 and maxWeight: 25 to see the progression quickly, then tighten your thresholds as you gain confidence.