kubectl exec for debugging

Have you ever deployed a container to Kubernetes, watched it crash-loop, and felt powerless because you couldn't see what was happening inside the pod? Logs alone often fall short — they show what was printed but not why a service refuses to start or a configuration file is missing. With kubectl exec, you can jump directly into a running container, poke around its filesystem, inspect environment variables, and even run diagnostic commands interactively — turning a black-box pod into a transparent debugging environment.

The problem this lesson solves

Kubernetes pods are designed to be ephemeral — they come and go based on replicas, node failures, or rolling updates. This ephemerality is great for resilience but horrible for debugging. When something goes wrong, you need to answer questions like:

• Did the container start with the correct environment variables?
• Is the configuration file present and well-formed?
• Can the container reach its database on the expected port?
• Why is my application returning 500 errors while the health check passes?

Logs (kubectl logs) give you application output, but they cannot probe the runtime environment. kubectl exec bridges this gap by letting you execute arbitrary commands inside a running container — just like SSH for Kubernetes pods.

Core concept / mental model

Think of kubectl exec as a remote shell for pods. Every container in a pod has its own filesystem, process tree, and network stack. When you run kubectl exec, you ask the Kubernetes API server to redirect your terminal session into one of those containers. The command runs with the same permissions as the container's entrypoint — not as a privileged superuser unless you configure security context.

Pro tip: kubectl exec requires the container to be running. If your pod is in CrashLoopBackOff, you cannot exec into it because the container process has already exited. In that case, use kubectl debug to spawn a temporary debugging sidecar (a topic for a later lesson).

Key terminology

• Pod: The smallest deployable unit in Kubernetes, containing one or more containers.
• Container: An isolated user-space instance within a pod.
• TTY (teletypewriter): An interactive terminal session that allows sending input and receiving output.
• STDIN / STDOUT: Standard input and output streams. Interactive commands (like /bin/bash) need both.

How it works step by step

When you issue kubectl exec -it <pod> -- <command>, here's what happens under the hood:

1. kubectl makes an API call to the Kubernetes API server with the pod name, namespace, and command.
2. The API server authenticates your request (using kubeconfig) and authorizes it with RBAC.
3. The API server forwards the request to the kubelet running on the node where the pod is scheduled.
4. The kubelet uses the container runtime interface (CRI) — typically containerd — to start the command inside the target container.
5. The command's STDIN and STDOUT are streamed back through the API server to your terminal.

If you omit the -i flag, STDIN is not attached; the command runs and exits immediately. If you omit the -t flag, no pseudo-TTY is allocated; commands expecting an interactive shell (like /bin/bash) will behave differently (usually exit immediately).

Blockquote best practice: Always use the full form kubectl exec -it <pod> -c <container> -- /bin/sh for maximum portability across base images.

Hands-on walkthrough

Prerequisites

• A running Kubernetes cluster (minikube, kind, or any cloud provider)
• kubectl configured to talk to your cluster
• A running pod to debug. For this exercise, we'll use the official nginx image.

Step 1: Deploy a sample pod

kubectl run debug-pod --image=nginx:alpine --port=80 --labels="app=debug"
kubectl get pods -l app=debug

Expected output (after a few seconds):

NAME        READY   STATUS    RESTARTS   AGE
debug-pod   1/1     Running   0          45s

Step 2: Exec an interactive shell

kubectl exec -it debug-pod -- /bin/sh

You are now inside the nginx container! Your terminal prompt changes to something like:

/ #

Step 3: Explore the container filesystem

From within the container, run:

ls -la /etc/nginx/
cat /etc/nginx/conf.d/default.conf
env | grep -i nginx
ps aux

Expected output (partial):

/ # cat /etc/nginx/conf.d/default.conf
server {
    listen       80;
    server_name  localhost;
    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }
}

/ # env | grep -i nginx
NGINX_VERSION=1.27.0

You just confirmed: - The configuration file is present and syntactically valid. - The environment variable NGINX_VERSION matches expectations. - The container is listening on port 80 inside the pod.

Step 4: Run a network diagnostic

Still inside the container:

wget -qO- http://localhost

Expected output (the default nginx welcome page, served as HTML).

Step 5: Exit the container

Type exit or press Ctrl+D to leave the interactive session.

Alternate syntax: Single command (non-interactive)

If you only need to check one thing, you can run the command inline:

kubectl exec debug-pod -- cat /etc/hostname

This outputs the pod's hostname (usually the pod name) and exits immediately — no interactive shell required.

Compare options / when to choose what

Choosing kubectl exec is the right call when: - The container is running and responsive. - You need interactive exploration (not just a single file). - The container image includes shell utilities like sh, curl, cat, ps.

If the container is minimal (e.g., a distroless image without a shell), you may need kubectl debug with an ephemeral container that carries its own toolbox.

Troubleshooting & edge cases

1. "Unable to use a TTY - container is not running or allocated a TTY"

Cause: You used -t but the container process did not allocate a TTY. This happens with non-interactive commands like kubectl exec -it pod -- ls. Solution: drop -t for non-interactive commands, or ensure you are running an actual shell.

2. "OCI runtime exec failed: exec failed: unable to start container process: exec: \"/bin/bash\": stat /bin/bash: no such file or directory"

Cause: Many container images (especially Alpine-based ones) do not include Bash. Use /bin/sh instead, which is POSIX and available in virtually every image.

3. Command hangs or never returns

Cause: The command is waiting for input. For example, kubectl exec debug-pod -- cat without arguments waits for stdin. Always pass the -i flag or provide arguments.

4. Permissions issues

Cause: The container runs as a non-root user and the command requires elevated privileges. You can inspect the security context with:

kubectl get pod debug-pod -o jsonpath='{.spec.containers[0].securityContext}'

If empty, the container runs as root. If you need to debug a restricted container, consider using kubectl debug or a debug sidecar with CAP_SYS_PTRACE.

5. "error: unable to upgrade connection: container not found ("app")"

Cause: The pod has multiple containers and you did not specify which one. Use the -c (or --container) flag:

kubectl exec -it debug-pod -c nginx -- /bin/sh

What you learned & what's next

You now know how to use kubectl exec for interactive debugging in Kubernetes. Specifically:

• Explain the core idea: kubectl exec grants a remote shell inside a running container, essential for diagnosing runtime issues that logs cannot reveal.
• Complete a practical exercise: You deployed a pod, exec'd into it, explored the filesystem, ran a network test, and verified configuration — all from the command line.
• Connect to the next lesson: In the next module, you'll learn kubectl debug, which extends this capability to pods that are crash-looping or running distroless images without a shell. You'll also explore ephemeral containers — a more powerful debugging primitive for production clusters.

Pro tip: Bookmark the kubectl exec cheat sheet: -it for interactive, -c for multi-container pods, -- to separate kubectl flags from the command. Your debugging speed will thank you.