Set up mTLS between services

Your Kubernetes cluster is running, pods are talking to each other, but any rogue service could eavesdrop or tamper with that traffic. Without mutual TLS (mTLS), your service mesh is a house with unlocked doors. This lesson shows you how to set up mTLS between services so every request is encrypted and both sides prove their identity before exchanging a single byte.

The problem this lesson solves

By default, network traffic inside a Kubernetes cluster is unencrypted plaintext. A compromised pod running in the same namespace, or even a malicious sidecar, can sniff or spoof service-to-service HTTP/gRPC calls. This is a critical gap for workloads handling PII, payments, or internal API keys.

Typical pain points without mTLS:

• No authentication — any pod can impersonate any other service.
• No encryption — data in transit is visible on the wire.
• Difficult compliance — standards like PCI-DSS, HIPAA, and SOC 2 require encrypted internal traffic.
• Manual complexity — distributing and rotating TLS certificates across dozens of microservices is error-prone.

Core concept / mental model

Think of mTLS as a two-way ID check at a secure facility. In standard HTTPS (server-side TLS), only the client verifies the server's certificate. With mTLS, both sides present their certificates and verify each other's signatures. If either identity is invalid, the connection is dropped.

Pro tip: mTLS is sometimes called "mutual authentication" or "two-way TLS." It's the foundation of zero-trust networking: never trust, always verify.

How it works in Kubernetes

A service mesh (e.g., Istio, Linkerd, or a CNI plugin) injects a sidecar proxy next to each pod. That sidecar:

1. Holds a certificate issued by a cluster-internal certificate authority (CA).
2. Intercepts all incoming and outgoing traffic for its pod.
3. Negotiates mTLS with the peer sidecar: exchange certificates, verify signatures, then encrypt the session.

Key components

How it works step by step

1. Choose a mesh or CNI — Istio and Cilium are popular options that support transparent mTLS.
2. Install the mesh — inject the control plane and configure sidecar injection for your namespaces.
3. Define a peer authentication policy — use a PeerAuthentication resource to set the mTLS mode.
4. Optionally restrict per-port — fine-tune which ports require mTLS vs. allow plaintext (useful for legacy probes).
5. Verify traffic is encrypted — deploy a test client and server, then inspect the logs and certificate details.

Hands-on walkthrough

This example uses Istio because it's the most widely adopted service mesh. If you don't have Istio installed, follow the Istio quickstart.

Prerequisites

• A Kubernetes cluster (minikube or kind is fine)
• istioctl CLI installed
• kubectl configured

Step 1: Install Istio with mTLS defaults

istioctl install --set profile=default -y

Verify the installation:

kubectl get pods -n istio-system

You should see pods like istiod, istio-ingressgateway, and istio-egressgateway (depending on profile).

Step 2: Enable automatic sidecar injection

Enable injection for a namespace (we'll use default):

kubectl label namespace default istio-injection=enabled

Step 3: Deploy a sample client and server

Create two deployments — one httpbin (server) and one sleep (client) — both with sidecars injected.

# httpbin.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        name: httpbin
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  ports:
  - port: 80
    targetPort: 80
    name: http
  selector:
    app: httpbin

# sleep.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "3650d"]
        imagePullPolicy: IfNotPresent

Apply both:

kubectl apply -f httpbin.yaml -f sleep.yaml

Wait for both pods to be ready (2/2 containers for injected pods).

Step 4: Apply a STRICT mTLS policy

By default, Istio runs in PERMISSIVE mode — it accepts both plaintext and mTLS. To enforce mTLS everywhere in the default namespace, create this PeerAuthentication:

# strict-mtls.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT

kubectl apply -f strict-mtls.yaml

Step 5: Verify mTLS is active

Exec into the sleep pod and make a request to httpbin:

kubectl exec -it deploy/sleep -- sh -c 'curl -s http://httpbin.default.svc.cluster.local/get'

You should see a normal JSON response. The magic is that the sidecars transparently upgraded the connection to mTLS. To confirm, check the logs of the httpbin sidecar:

kubectl logs deploy/httpbin -c istio-proxy | grep "mutual"

Look for entries like:

"authority": "httpbin.default.svc.cluster.local:80",
"requested_server_name": "outbound_.80_._.httpbin.default.svc.cluster.local",
"upstream_cluster": "inbound|80|http|httpbin.default.svc.cluster.local",
"response_code": "200",
"downstream_tls_version": "TLSv1.3",
"downstream_peer_cert": "CN=spiffe://cluster.local/ns/default/sa/default"

Compare options / when to choose what

Pro tip: Start with PERMISSIVE mode, monitor traffic for breakages, then switch to STRICT after verifying nothing relies on plaintext.

Troubleshooting & edge cases

Connection refused after enabling STRICT

If existing clients don't have a sidecar, they can't upgrade to mTLS. Switch to PERMISSIVE and ensure all client pods have sidecars injected.

kubectl delete peerauthentication default -n default
kubectl label namespace default istio-injection=enabled --overwrite
kubectl rollout restart deploy/sleep

Health checks failing

Kubelet probes (liveness, readiness) don't go through the sidecar. Istio automatically avoids mTLS for port 10250 and 15021. If using a custom health check port, add an exception:

spec:
  portLevelMtls:
    8080:
      mode: DISABLE

Certificate expiration or rotation errors

By default, Istio rotates certificates every 24 hours. If your cluster clock skew exceeds 5 minutes, rotation may fail. Verify with:

kubectl get secret -n istio-system istio-ca-secret -o yaml | grep "ca-cert"

What you learned & what's next

You now understand how to set up mTLS between services using a service mesh like Istio. You can explain the problem of unencrypted service-to-service traffic, apply a PeerAuthentication policy, verify that communication is encrypted and authenticated, and troubleshoot common issues like broken health checks or strict-mode connection failures.

This is a stepping stone into zero-trust networking and service mesh security. Next, you'll learn how to implement fine-grained authorization policies using AuthorizationPolicy resources to control which services can talk to which endpoints.