The Evolution of Kubernetes Networking: From Iptables to eBPF

In the early days of Kubernetes, getting a simple web app to talk to a database felt like wrestling a hydra. Pods came and went, IPs were ephemeral, and DNS was a luxury. Seven years later, the networking stack has evolved from a messy tangle of iptables rules into a sophisticated ecosystem of service meshes, eBPF-powered data planes, and declarative APIs. Here’s how we got here—and where we’re going.

The CNI Revolution

Kubernetes’ original networking model was elegant on paper: every pod gets a unique IP, no NAT required. The problem? Actually implementing this at scale was a nightmare. The Container Network Interface (CNI) standard, introduced in 2016, was the first major leap. It turned networking into a plug-and-play game.

Suddenly, you could swap out Flannel for Calico, or Weave for Cilium, without touching your cluster config. CNIs handled IP assignment, routing, and simple policies. But they were blissfully ignorant of what applications were actually doing—they just moved packets. This was fine for basic microservices, but it left teams solving the same discovery and security problems manually.

Service Discovery: From Kube-Proxy to Native DNS

The built-in service discovery mechanism was a stroke of genius: a Service object gets a stable IP (ClusterIP) and a DNS name (my-svc.namespace.svc.cluster.local). Under the hood, kube-proxy used iptables to rewrite packets from the service VIP to a random pod.

It worked—until it didn’t at scale. Iptables rules are O(n) in the number of services, and clusters with 10,000+ services would grind to a halt. The response was IPVS (IP Virtual Server), which uses hash tables instead of chains. Suddenly, the same load balancing logic ran in O(1), and kube-proxy stopped being a bottleneck.

But the bigger shift came from within: CoreDNS replaced kube-dns as the default in Kubernetes 1.11. It wasn’t just faster—its plugin architecture allowed custom resolvers, DNS-over-HTTPS, and zone transfers. Teams could finally query services by name, get weighted round-robin results, and even do external name resolution.

The Ingress Explosion (and Its Aftermath)

The original Ingress resource was a Swiss Army knife—and about as awkward to use. It handled HTTP routing with YAML annotations that varied wildly between controllers (nginx, Traefik, HAProxy). Everyone had a different conversation about TLS termination paths or sticky sessions.

The real breakthrough came with the Gateway API in 2021. This was a clean break: instead of one monolithic resource, you got GatewayClass, Gateway, HTTPRoute, TCPRoute, and a dozen other composable objects. Now you could say “route traffic for example.com/api to this backend” without caring whether the controller was Envoy, Contour, or Istio. It also solved the problem of multi-team clusters—each team got their own Route, while the platform team owned the Gateway.

Service Mesh: The Maturity Pivot

Around 2018–2019, every blog post was “Why you need a service mesh.” The promises were seductive: mTLS everywhere, circuit breakers, retries, observability. Istio and Linkerd delivered these—but with a cost. The sidecar proxy (Envoy, mostly) added 10–30ms of latency per hop, consumed gigs of memory per node, and debugging meant staring at proxy logs you didn’t write.

The industry pivoted hard. Istio introduced the ambient mesh in 2022, ditching sidecars for per-node proxies. Instead of injecting a sidecar into every pod, it ran a shared proxy on each node, reducing resource overhead by 70–90%. Linkerd2 took a different route—optimized the Rust-based proxy to run in single-digit millisecond overhead.

The real winner? Simplify don’t solve. Teams realized they didn’t need mTLS for every internal call—they needed it for PCI-billed traffic and cross-cluster communication. The mesh became a selective tool, not a blanket.

eBPF: The Game Changer Nobody Saw Coming

eBPF (extended Berkeley Packet Filter) started as a Linux kernel curiosity—a way to run sandboxed programs inside the kernel. What happened next changed networking fundamentally.

Cilium, the eBPF-based CNI, showed that you could replace iptables, kube-proxy, and even Envoy sidecars with kernel-level programs. No deep packet inspection overhead. No proxy context switches. A single eBPF program can filter, forward, encrypt, and observe traffic—and it runs in nanoseconds.

• No more kube-proxy: Cilium can do direct server return (DSR), bypassing the NodePort bottleneck.
• Identity-based security: Instead of IP whitelists, use Kubernetes service account names or labels. eBPF tracks identity at the kernel level.
• Real observability: Hubble (built on Cilium) shows you every TCP handshake, latency spike, dropped packet—without instrumenting applications.

But eBPF isn’t a silver bullet. It requires Linux 5.10+ (RHEL 8.6, Ubuntu 22.04), and debugging a misbehaving eBPF program is like debugging a crashed kernel module. Still, the performance wins are undeniable: Facebook (Meta) cut their load balancer infrastructure in half by using eBPF.

The Cilium/Istio Power Struggle

Two projects now compete for the “default networking” crown: Cilium and Istio (with ambient). Cilium claims eBPF makes sidecars obsolete; Istio argues that some control plane logic (retries, circuit breakers) is inherently complex and belongs in a proxy.

The truth is, they’re converging. Cilium now supports sidecar injection for Istio—calls it “Tetragon.” Istio migrated its data plane to use Cilium’s eBPF for kernel-level routing. The winner isn’t a specific project—it’s the interface: both expose the same Gateway API and sidecar-less operation. By 2024, most new clusters run either Cilium with eBPF or Istio ambient, and neither uses iptables anymore.

Service Mesh 2.0: No Sidecars

The biggest architectural shift since 2023 is the death of the sidecar proxy—at least, the death of mandatory sidecars. Both Istio and Linkerd now offer “sidecarless” modes:

• Istio ambient: Deploy a per-node ztunnel (zero-trust tunnel) component that handles mTLS, authentication, and L4 policies. An optional Waypoint proxy per-namespace handles L7 features like retries.
• Linkerd mesh: Weaveworks’ Flagger and Linkerd combine to use the kernel’s built-in WireGuard for encryption, avoiding proxied transitions.

The result? Pods that don’t need a second container just to talk to each other. Memory overhead dropped from 100MB+ per proxy to <10MB per node.

What’s Next: Serverless Networks and eBPF-as-a-Service

The next frontier is the fully serverless network: no cluster-node boundaries, no VPC tunnels, no NAT. This is already happening through:

• Cloud Native Network Functions (CNFs): Telcos are running 5G core functions on Kubernetes with eBPF-based routing, replacing hardware routers.
• Multi-cluster Service Mesh: Projects like Submariner and ClusterLink let two clusters share a single service mesh across AWS/GCP/on-prem, with eBPF handling traffic steering.
• eBPF-as-a-Service: A startup called “eBPF Enterprises” now offers a managed eBPF sandbox—upload a C program, get it deployed across your fleet, and observe its execution.

The old world of tweaking iptables rules in a configmap is dead. The new world is about writing eBPF programs that understand your app’s intent—and running them at kernel speed. What hasn’t changed is the fundamental principle: Kubernetes networking is still about connecting decoupled services. The tools just got a lot smarter.