Kubernetes Pod Networking

You’ve got a Python service containerized on Kubernetes. Pods are running, Deployments are happy, but suddenly another service can’t reach it — or your database pod is unreachable from your API pod. Networking in Kubernetes can feel like a black box, and when things break, the debug loop is painful. In this lesson, you’ll go beyond kubectl expose and understand exactly how pod networking works for Python services — from the flat network model to CNI plugins, service discovery, and the tools you can use to debug. By the end, you’ll be able to reason about your cluster’s network, diagnose failures, and make confident architecture choices.

The problem this lesson solves

You have a microservices architecture with Python services — FastAPI, Django, perhaps a Celery worker. All of a sudden, a service times out when calling another service. Or maybe a pod can ping the cluster IP but can’t reach the service DNS name. Networking issues rank high among the most confusing problems for Python developers moving to Kubernetes.

• You spent time writing Dockerfiles and YAML manifests, but you never really understood how Pods get IP addresses or how traffic flows.
• You’ve heard of ClusterIP, NodePort, and LoadBalancer, but you don’t know when to use which — or why a service DNS name sometimes just doesn't resolve.
• You think of networking in terms of containers, but Kubernetes abstracts that into a flat network where every Pod gets its own IP and can talk to any other Pod directly — but only if the network layer is correctly configured.

Why now? As you scale Python services, you’ll rely on internal service-to-service communication (often via HTTP/gRPC). A solid mental model of pod networking prevents head-scratching outages and makes you faster at debugging.

Core concept / mental model

Kubernetes does not set up its own networking by default. Instead, it expects a Container Network Interface (CNI) plugin to implement a flat network model. The fundamental rules of pod networking are:

1. Every Pod gets its own unique IP address.
2. Pods can communicate with all other Pods without NAT (no address translation) — they can reach each other directly.
3. Agents (like kubelet, or the kube-proxy) on each node communicate with all Pods on that node — and can reach them via that Pod IP.

Think about it like this: each Pod is a little host with its own NIC. Your Python app inside the container sees eth0 with an IP like 10.244.0.5. That IP is routable across the entire cluster, regardless of which node the Pod runs on.

The CNI plugin (like Calico, Flannel, or Cilium) is responsible for:

• Allocating IP addresses to Pods.
• Setting up virtual interfaces (veth pairs) that connect each Pod's network namespace to the node's network.
• Implementing routing rules so traffic between Pods on different nodes can flow correctly.

Why does this matter for Python services?

Your Python service doesn't need to know about CNI at all — it just binds to a port (like 8000) and listens. Kubernetes's networking layer makes sure it can be reached at that Pod IP. But when you expose your service via a Service object, kube-proxy (or a CNI's service mesh) implements virtual IPs and load balancing. Understanding this layering is what separates a dev who guesses from an engineer who debugs.

How it works step by step

Let's trace what happens when another Pod (say, your frontend service) sends a request to your Python API service.

1. Pod-to-Pod communication (direct): If the frontend pod knows the API pod's IP, it can send traffic directly — the kernel routes it through the node's routing table. The packet exits the pod's network namespace via a veth pair to the node's root namespace, then hops across nodes (or within the same node) to the destination pod. The CNI plugin handles that routing.

2. Service abstraction: Typically, you don't use Pod IPs directly — Pods are ephemeral. You create a Service object with a stable ClusterIP. The service has a label selector that matches your Python pods' labels.

3. kube-proxy watches the API server: For each Service, kube-proxy programs iptables rules (or IPVS rules). These rules intercept traffic destined to the Service's ClusterIP and forward it to one of the selected Pods' IPs (using a random algorithm). So when your frontend calls http://api-service:8000, the kernel redirects the packet as per the iptables rules on that node.

4. DNS resolution: The cluster's DNS service (CoreDNS) resolves the service name to its ClusterIP. Your Python app can use standard Python HTTP clients — like requests or httpx — and they’ll resolve the service name automatically because the container’s /etc/resolv.conf points to CoreDNS.

5. Response path: The response packet goes back through the same path, but thanks to conntrack (connection tracking), the reverse NAT is applied correctly, so the frontend sees the response as coming from the Service IP.

The role of network policies

By default, all Pods can talk to each other. But in production, you want to restrict traffic. NetworkPolicies are Kubernetes resources that define which Pods can communicate. They are implemented by CNI plugins like Calico or Cilium — they enforce the rules at the interface level. This is crucial for a multi-tenant or security-sensitive environment.

Hands-on walkthrough

Let’s put this into practice. You’ll deploy two Python services: a simple API that echoes a message, and a test client that calls it. You’ll see how pod IPs are assigned, how services work, and how the network behaves.

Step 1: Deploy a simple Flask/FastAPI service

Create a file echo-api.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo-api
  template:
    metadata:
      labels:
        app: echo-api
    spec:
      containers:
      - name: echo-api
        image: hashicorp/http-echo:latest
        args: ["-text=\"hello from echo-api\""]
        ports:
        - containerPort: 5678

Apply it:

kubectl apply -f echo-api.yaml

Check that pods are running and get their IPs:

kubectl get pods -o wide

Expected output (IPs will differ):

NAME                        READY   STATUS    RESTARTS   AGE   IP           NODE
 echo-api-6b5f8d9f4-abc123 1/1     Running   0          10s   10.244.0.7   node1
 echo-api-6b5f8d9f4-def456 1/1     Running   0          10s   10.244.0.8   node1

Step 2: Create a service for it

Create echo-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: echo-api-svc
spec:
  selector:
    app: echo-api
  ports:
    - port: 80
      targetPort: 5678

Apply and check the ClusterIP:

kubectl apply -f echo-service.yaml
kubectl get svc echo-api-svc

Output:

NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
echo-api-svc  ClusterIP   10.96.120.50    <none>        80/TCP    10s

Step 3: Deploy a Python test client

Create a simple Python pod that uses requests to call the service by name:

apiVersion: v1
kind: Pod
metadata:
  name: test-client
spec:
  containers:
  - name: client
    image: python:3.12-slim
    command: ["sh", "-c", "pip install requests && python -c \"import requests; print(requests.get('http://echo-api-svc').text)\""]

Apply and check logs:

kubectl apply -f client-pod.yaml
kubectl logs test-client

Expected output:

hello from echo-api

This proves that service discovery works: the container resolved echo-api-svc to the ClusterIP and got a response from one of the echo pods.

Step 4: Inspect the network with tcpdump (optional)

You can get a Pod shell and use tcpdump to see traffic. First, install tcpdump inside the test client (or use a debug container):

kubectl exec -it test-client -- bash
apt-get update && apt-get install -y tcpdump

Then run a background HTTP call and watch packets:

tcpdump -n -i eth0 port 80 &
curl http://echo-api-svc

You'll see packets captured with source and destination IPs — that's your service's ClusterIP, and behind the scenes, the real destination is one of the pod IPs.

Compare options / when to choose what

When to choose what:

• Internal API calls: Use a ClusterIP service. It's the default for a good reason.
• Exposing a service to the internet in production: Use a LoadBalancer (or an Ingress, which is the next lesson in this track).
• Stateful workloads like PostgreSQL: Use a headless service so each pod gets a stable DNS name (e.g., postgres-0.postgres-svc.default.svc.cluster.local).
• If you need strict security: Define NetworkPolicies to lock down communication between pods.

Troubleshooting & edge cases

Even with a correct setup, things can fail. Here are the most common pitfalls and how to fix them:

1. Service DNS doesn't resolve

Symptom: curl http://echo-api-svc returns Name or service not known.

Check: Are you in the same namespace? Service names must include the namespace if different. Try curl http://echo-api-svc.default.svc.cluster.local. Also verify that the CoreDNS pods are running.

kubectl get pods -n kube-system -l k8s-app=kube-dns

If CoreDNS isn't healthy, you won't get DNS at all.

2. Traffic times out when hitting the ClusterIP

Symptom: You can ping a pod IP, but curl to the service times out.

Resolution: Check if the service selector matches the pod labels. Use kubectl get endpoints <service> to see if the service has endpoints. If the endpoints are empty, your label selector is wrong.

kubectl get endpoints echo-api-svc

If it’s empty, edit your service to fix the selector.

3. Pods can’t communicate across nodes

Symptom: Pods on node A can talk to each other, but can’t reach pods on node B.

Cause: The CNI plugin isn’t configured correctly for multi-node networking. Check if all nodes are ready, and inspect the CNI plugin’s logs:

kubectl get pods -n kube-system | grep -E 'calico|flannel|weave'
kubectl logs -n kube-system <cni-pod>

If using Flannel, ensure each node has a flannel.1 interface. In many setups, the default CNI works out of the box, but cloud-specific configurations can break.

4. Python service not listening on the right interface

Symptom: You set containerPort: 8000 in YAML, but your Python app listens only on 127.0.0.1. Pod networking requires your app to bind to 0.0.0.0 (or the container’s eth0 IP).

Fix: Make sure your Flask/FastAPI app runs with app.run(host="0.0.0.0") or use uvicorn --host 0.0.0.0. Otherwise, the kube-proxy can't forward traffic to your pod.

5. NetworkPolicy blocking traffic silently

Symptom: Services worked before, but after adding a NetworkPolicy, traffic stops.

Check: Review your NetworkPolicy rules. Remember, the default deny-all blocks everything unless explicitly allowed. Use kubectl get networkpolicy to see active policies.

Pro tip: Always test with a simple curl from a debug pod. If you can't reach the pod IP but the pod is running, it's usually a CNI or NetworkPolicy issue, not your Python code.

What you learned & what's next

You now have a solid mental model of pod networking in Kubernetes for Python services. You understand:

• Every pod gets an IP on a flat network, using a CNI plugin
• How Services (ClusterIP, NodePort, LoadBalancer) route traffic to pods
• How DNS and kube-proxy work together for service discovery
• How to debug common networking failures

What’s next: The next lesson in this track dives into Ingress — how to expose your Python services to the outside world elegantly, with path-based routing and TLS termination. You’ll use the same networking knowledge but add a layer of external access. Get ready to put an Ingress controller in front of your API!

Remember: networking in Kubernetes is not magic. It’s a set of layers. Now that you understand the layers, you can debug with confidence instead of guessing.