Canary Releases with Traffic Splitting

Your last deployment broke the checkout page, and you only found out after 10,000 users had already seen the error. You roll back fast, but the damage is done—and your team loses trust in shipping. If that scenario makes you wince, you're ready for canary releases with traffic splitting, a strategy that lets you ship new versions to a small slice of users first, validate in production, then ramp up with confidence. This lesson shows you how to implement canary releases with traffic splitting in your CI/CD pipeline, step by step, so you can deploy boldly without breaking the world.

The problem this lesson solves

Traditional deployments are all-or-nothing. You push a new version to every server, and if it's broken, every user is affected. Even with staging environments, you never truly replicate production—traffic patterns, data volume, and real user behavior are unique. The result is fear: fear of deploying, long freezes, and emergency rollbacks that erase progress. Canary releases solve this by shrinking the blast radius. You send a small percentage of traffic to the new version, watch it closely, and only scale up when you're confident. This lesson teaches you to implement canary releases with traffic splitting, turning deployment from a gamble into a measurement.

Core concept / mental model

Think of canary releases like the old mining practice: a canary in a coal mine. If the canary dies, miners know the air is toxic before it kills the crew. In software, the canary is your new version, and the mine is your production traffic. You expose the new version to a tiny subset of users—say 5%—and monitor for signs of trouble. If all looks good, you gradually increase the percentage. If something breaks, you redirect that 5% back to the stable version, affecting almost nobody.

Traffic splitting is the mechanism that makes canary releases possible. It's not about routing specific users to a version; it's about proportionally dividing incoming requests between the old and new versions. Every user has a chance of hitting the new version, but only the canary percentage actually does. Tools like Kubernetes (with a service mesh like Istio or Linkerd), Nginx, or a load balancer can perform this split based on weights. The key insight: you're not choosing who gets the new version; you're deciding what fraction of requests do. This is continuous delivery's answer to "test in production"—it's structured, measurable, and reversible.

How it works step by step

Implementing canary releases with traffic splitting follows a repeatable sequence. Each step builds on the last, and the whole cycle fits inside your CI/CD pipeline.

1. Deploy the stable version — Ensure the current version (v1) runs at full capacity. This is your baseline.
2. Deploy the canary version — Create a second instance of the deployment (v2) with the same resources. It receives zero traffic initially; it's just sitting there.
3. Configure traffic splitting — Set up a router (e.g., an Ingress controller, service mesh, or load balancer) with weighted routing. Start with a small percentage (e.g., 5%) pointing to v2.
4. Monitor health metrics — Track error rates, latency, and business metrics (like conversion) for both versions. The canary's performance vs. the stable version indicates health.
5. Gradually increase the canary's share — If metrics are healthy, ramp up in stages: 10%, 25%, 50%, 100%. Each stage should run for a cooldown period to collect enough data.
6. Promote or rollback — At 100%, promote v2 to stable (update the primary deployment) and teardown v1. If at any point the canary shows problems, drop its traffic to 0% immediately, effectively rolling back.

This sequence turns deployment into a controlled experiment, with each ramp-up a hypothesis test. The key is that rollback is as simple as changing a weight—not redeploying an old artifact.

Hands-on walkthrough

Let's put this into practice with a concrete example. We'll use Kubernetes with Nginx Ingress as our traffic splitter, since it's a common, accessible stack. But the principles apply to any tool. First, create the stable deployment and service:

# stable.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-v1
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: v1
  template:
    metadata:
      labels:
        app: myapp
        version: v1
    spec:
      containers:
      - name: app
        image: myapp:1.0.0
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
    version: v1
  ports:
  - port: 80
    targetPort: 8080

Apply it:

kubectl apply -f stable.yaml

Next, deploy the canary version as a separate deployment but with the same service labels, so it can receive traffic:

# canary.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
      version: v2
  template:
    metadata:
      labels:
        app: myapp
        version: v2
    spec:
      containers:
      - name: app
        image: myapp:2.0.0
        ports:
        - containerPort: 8080

kubectl apply -f canary.yaml

Now, create an Ingress with NGINX's built-in canary annotation. This annotation splits traffic based on weight:

# canary-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10" # or "5"
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp
            port:
              number: 80

Apply it, and check the ingress status:

kubectl apply -f canary-ingress.yaml
kubectl get ingress

Now, 10% of traffic to example.com goes to the canary service (which selects v2 pods). To verify, use a load generator and watch the logs:

for i in {1..100}; do curl -s http://example.com/ | grep version; done | sort | uniq -c

Expected output (approximate):

10 version: 2.0.0
90 version: 1.0.0

When you're confident, update the canary weight to 50%, then 100%:

kubectl annotate ingress myapp-canary nginx.ingress.kubernetes.io/canary-weight="50" --overwrite

Finally, when switching fully to v2, update the main service selector to point to version: v2 and delete the canary ingress:

kubectl delete ingress myapp-canary

Compare options / when to choose what

Several tools implement traffic splitting. Choose based on your platform and complexity needs. The table below compares common options.

Pro tip: Start with Nginx Ingress if you're on Kubernetes and just want the pattern down. Move to a service mesh when you need richer traffic policies or automated canary analysis.

The choice also depends on your release cadence—if you ship daily, automated canaries (Flagger) save time; if weekly, manual weight changes are fine.

Troubleshooting & edge cases

Canary releases fail in predictable ways. Here are the common pitfalls and how to fix them.

• No traffic splits as expected. Check that the canary Ingress has the correct service selector. The service must match the canary deployment's labels. Also ensure the canary annotation is exactly nginx.ingress.kubernetes.io/canary: "true" (quotes matter).
• Sessions break during canary. If your app stores state in memory, users hitting both versions see inconsistent behavior. Use sticky sessions (session affinity) or migrate to a shared session store. In Nginx Ingress, add nginx.ingress.kubernetes.io/affinity: "cookie".
• database schema conflicts. When v2 changes the schema, v1 and v2 hitting the same DB cause errors. Ensure backward-compatible migrations or split the canary to run against a separate DB for testing.
• Metrics show false positives. Small traffic volume (5%) may not trigger monitoring thresholds. Lower the alert threshold for the canary or run for a longer time.
• Rollback is not instant. Some requests already in flight will hit the canary. Choose a timeout that accepts this. With Nginx, you can unapply the canary Ingress—it removes immediately.
• Canary sits too long. If you never ramp up, you're just blue-green. Set a time-based promotion policy or automated analysis.

Pro tip: Always monitor both versions. If the canary is better, promote quickly; if worse, rollback. The decision should be data-driven, not emotional.

What you learned & what's next

You now understand the core idea behind implementing canary releases with traffic splitting—incremental exposure reduces risk and accelerates deployment. You completed a hands-on exercise using Nginx Ingress to split 10% traffic to a canary deployment, monitored it, and promoted it to full production. You learned to compare tools like Nginx, Istio, Linkerd, and Flagger, and you know how to troubleshoot common issues like session affinity and database migrations.

Next, you'll explore automated rollback and progressive delivery in the CI/CD foundations track. You'll learn how tools like Flagger automate canary analysis and rollback, making the process hands-off. That lesson will build directly on this one—so keep your canary pipeline ready.