Watch Kubernetes Events with Python

If you've ever watched a pod crash-loop, then deleted, recreated, and watched it crash again — all while checking kubectl get events every few seconds — you know the pain of manual cluster monitoring. Kubernetes generates a constant stream of events about scheduling, health checks, and resource issues, but relying on kubectl means you're always one step behind. In this lesson, you'll learn how to watch Kubernetes events with the Python client and turn that firehose of data into automated reactions, real-time dashboards, or alerting systems.

The problem this lesson solves

Manual event monitoring is reactive, error-prone, and doesn't scale. When a Deployment is rolling out or a node is under pressure, events appear and disappear in seconds. By the time you run kubectl get events —watch, the root cause may already be gone, leaving you with nothing but a mystery and a baffled SRE.

Python automation solves this. Instead of polling, your script can subscribe to the Kubernetes event stream in real time, react instantly to failures, and log historical events for post-mortem analysis. Whether you're building a custom alerting bot, a CI/CD pipeline that validates rollout health, or a simple heartbeat monitor, the ability to watch events programmatically is a superpower.

Core concept / mental model

Think of the Kubernetes API as a live sports commentator. kubectl get events is like reading the post-game report — you get the highlights but late. Watching events with the Python client is like tuning into the live broadcast: you hear every pitch, every foul, and every victory the moment it happens.

Technically, the watch mechanism works by opening a long-lived HTTP connection to the API server and streaming JSON-encoded events. Each event object has a type (e.g., ADDED, MODIFIED, DELETED) and the payload is a Kubernetes Event resource that includes details like reason (e.g., FailedScheduling, BackOff), message, involvedObject, and timestamp.

In the Python client, you use watch.Watch() to create a stream. You specify the API, such as list_namespaced_event for events in a specific namespace or list_event_for_all_namespaces for cluster-wide events. Under the hood, the client handles pagination, HTTP timeouts, and reconnection logic — so you can focus on what to do with the events, not how to fetch them.

How it works step by step

1. Set up a Kubernetes client — load the kubeconfig file or use in-cluster configuration (when running inside a pod).
2. Choose the event source — decide whether to watch all namespaces or a single namespace.
3. Create a watcher — instantiate watch.Watch() from the kubernetes.watch module.
4. Stream events — call .stream() with the API method and optional filters (like field_selector or namespace).
5. Process event objects — each event dictionary contains type and object (the event resource).
6. Handle termination — break the loop with a KeyboardInterrupt or timeout.

Pro tip: Always close the watcher (using finally or a context manager) to close the HTTP connection and avoid leaked resources.

Hands-on walkthrough

Prerequisites

Ensure you have the kubernetes pip package installed and a kubeconfig that points to your cluster:

pip install kubernetes
kubectl cluster-info

If you're running this from within a pod, the client can auto-discover the service account and namespace.

Example 1: Watch events in a specific namespace

This script streams events from the default namespace and prints a summary for each event.

from kubernetes import client, config, watch

config.load_kubeconfig()  # or config.load_incluster_config() inside a pod

v1 = client.CoreV1Api()
w = watch.Watch()

try:
    for event in w.stream(v1.list_namespaced_event, namespace="default"):
        eve = event["object"]
        print(f"{event['type']} {eve.reason}: {eve.message}")
        if eve.involved_object:
            print(f"  involved: {eve.involved_object.kind} {eve.involved_object.name}")
finally:
    w.stop()

Expected output (truncated):

ADDED Started: Started container nginx
ADDED Scheduled: Successfully assigned default/nginx-7c7f5d9b4c-r5v2l to minikube
MODIFIED Killing: Stopping container nginx

Example 2: Watch events for all namespaces with a timeout

Use a timeout_seconds when you only want to monitor for a short window, and add a field_selector to filter by resource name.

from kubernetes import client, config, watch

config.load_kubeconfig()

v1 = client.CoreV1Api()
w = watch.Watch()

field_selector = "involvedObject.name=my-pod"  # optional filter

try:
    for event in w.stream(
        v1.list_event_for_all_namespaces,
        timeout_seconds=30,
        field_selector=field_selector,
    ):
        eve = event["object"]
        print(f"[{event['type']}] {eve.reason} in {eve.namespace}: {eve.message}")
finally:
    w.stop()

Example 3: Build a simple event logger with error handling

For production, you'll want to reconnect on errors and log events to a file or a monitoring system.

import logging
from kubernetes import client, config, watch, ApiException

config.load_incluster_config()  # or load_kubeconfig()

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

v1 = client.CoreV1Api()
w = watch.Watch()

while True:
    try:
        for event in w.stream(v1.list_event_for_all_namespaces):
            eve = event["object"]
            logger.info("%s %s: %s", event["type"], eve.reason, eve.message)
    except ApiException as e:
        if e.status == 410:
            logger.warning("Watch expired, restarting.")
            continue
        raise
    finally:
        w.stop()

Compare options / when to choose what

For Python developers, the Python client is almost always the right choice when you need to react programmatically. For a one-off investigation, kubectl is faster. If you're running a fleet of clusters and want centralized event storage, a dedicated exporter may save time.

Note: Variants like list_namespaced_event vs. list_event_for_all_namespaces are just different scoping options; use namespaced watches when you only care about specific workloads to reduce noise.

Troubleshooting & edge cases

• Watch exits immediately — Often caused by an invalid field_selector. Double-check the syntax (e.g., involvedObject.name=foo) and that the resource exists.
• 403 Forbidden error — Your service account or kubeconfig user lacks list and watch permissions on events. Create a ClusterRole with rules: [{apiGroups: [""], resources: ["events"], verbs: ["list", "watch"]}].
• HTTP 410 Gone — The watch resource version is too old. Catch this error and restart the stream (as in Example 3).
• Missed events during restart — If you need zero loss, use a resource_version or resource_version_match to resume from a known point.
• High CPU/memory usage — If you're watching all namespaces with a high event rate, consider filtering with field_selector or namespace.

What you learned & what's next

You now know how to watch Kubernetes events with the Python client: you can set up a watcher, filter events, handle reconnects, and integrate event streams into your own automation. This is a foundational building block for building operators, health checkers, and reactive control loops.

In the next lesson, you'll learn how to respond to those events by creating or modifying Kubernetes resources programmatically — turning your watcher into a full-fledged custom controller.

Pro tip: Combine the event watcher with a threading.Timer to send a Slack alert after repeated BackOff events — a powerful pattern for self-healing applications!