Python Operator for Custom Resources

You’ve mastered Deployments, Services, and ConfigMaps — but what happens when Kubernetes doesn’t have a built-in resource for your domain-specific need? Maybe you need a PostgresCluster that spins up a pet and a headless service, or a CronJob-like BackupJob that also cleans up old volumes. Without a custom controller, you’re stuck writing shell scripts and glue code that no one can audit. This lesson shows you how to build a Python operator for custom resources — the Kubernetes-native way to encode your operational knowledge into the control plane itself. With the kopf framework, you’ll turn a Python function into a controller that watches your Custom Resources and reconciles the cluster to the desired state, and you’ll test it with minikube in under an hour.

The problem this lesson solves

Kubernetes is declarative: you write a YAML file, apply it, and the control plane makes it real. But that magic only covers built-in resources like Deployments and Services. When your app needs a custom abstraction — a database cluster, a scheduled backup, a per-team environment — there’s no kind: BackupJob by default. Without an operator, you’re forced to:

• Run manual kubectl commands that other team members must memorize.
• Write external cron scripts that fail silently and never retry.
• Manage state with spreadsheets or shared documents.

This is the classic operator pattern gap: the gap between "the user's desired state" and "the infrastructure that actually runs." The pain is real because modern Python microservices often need custom lifecycle logic — for example, when a new tenant signs up, you must create a Namespace, a Secret, and a ServiceAccount. Doing that by hand is error-prone and impossible to scale.

A Kubernetes operator fills that gap by encoding your operational knowledge as code. It watches Custom Resources (CRs), reads the declared desired state, and takes action to make the cluster match. Once built, your operator becomes the single source of truth for that resource type — just like the built-in controllers that run your Deployments.

Core concept / mental model

The heart of a Kubernetes operator is the reconciliation loop (often called the control loop). Think of a thermostat:

• You set the desired temperature (the spec).
• The thermostat senses the current temperature (the status).
• If they differ, it turns the heater on or off (the action).
• It repeats forever, reacting to any change.

In Kubernetes, the same loop runs as a Pod inside your cluster. Here’s the key vocabulary you’ll see in every operator:

• Custom Resource (CR): an instance of your new API type, defined by a YAML manifest. For example, a MyApp CR with spec.replicas: 3.
• Custom Resource Definition (CRD): the schema for your resource type. It tells Kubernetes what fields are allowed and how to store instances.
• Controller: the loop that watches CRs and compares desired state (spec) with actual state (status/live cluster objects).
• Operator: a controller plus domain-specific logic — the brain that knows how to cr年間 best practices, e.g., how to scale an app or send an alert.
• Reconcile: the action taken to move from current state to desired state.

A Python operator uses the kopf (Kubernetes Operator Framework) library. Under the hood, kopf uses the Kubernetes API to watch for events (add, update, delete) on your CRs and automatically calls your Python functions. You don’t write the loop yourself — you write what to do when something changes, and kopf handles the watching, retries, and timing.

Why Python and kopf?

Python is ideal for operators because of its readability and rich ecosystem. The kopf framework gives you decorators like @kopf.on.create and @kopf.on.update, so your code reads like a microservice handler. You keep the power of the Kubernetes API via the official kubernetes Python client, but you get a high-level interface that hides the complexity of watch streams and leader election.

How it works step by step

Building an operator isn't magic. Here’s the high-level sequence you’ll follow:

1. Define the CRD — a YAML file that tells Kubernetes about your new resource type (e.g., kind: Website).
2. Write the operator logic — a Python script that reacts to CR events and creates/manages related standard resources (e.g., a Deployment for your Website).
3. Package the operator — containerize it with Docker.
4. Deploy the operator to your cluster — usually as a Deployment, plus a ServiceAccount with RBAC permissions.
5. Test and observe — create a CR instance and watch the operator react.

The cause and effect are straightforward: When you kubectl apply a CR, the API server stores it. The operator (running in a Pod) gets a watch event, calls your @kopf.on.create function, which then uses the Kubernetes API to create a Deployment. When you delete the CR, the operator cleans up the related resources.

Hands-on walkthrough

Let’s build a real operator. We’ll create a Website CR that deploys a simple Nginx static site. You’ll need kubectl, minikube, Python 3.10+, and kopf installed. This walkthrough follows the official kopf tutorial but tightens it for Python developers.

1. Define the Custom Resource Definition (CRD)

First, create a file called crd.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: websites.example.com
spec:
  group: example.com
  names:
    kind: Website
    listKind: WebsiteList
    plural: websites
    singular: website
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                image:
                  type: string
                port:
                  type: integer

Apply it with kubectl apply -f crd.yaml. Now Kubernetes knows about kind: Website.

2. Write the operator logic

Create operator.py with this content:

import kopf
import kubernetes
import yaml

# Initialize the kubernetes client
kubernetes.config.load_incluster_config()
api = kubernetes.client.AppsV1Api()

@kopf.on.create('example.com', 'v1', 'websites')
def create_website(body, spec, **kwargs):
    """Create a Deployment for the Website CR.

    This is called when a Website resource is created.
    """
    name = body['metadata']['name']
    namespace = body['metadata'].get('namespace', 'default')
    image = spec.get('image', 'nginx:1.25')
    port = spec.get('port', 80)

    # Define the Deployment manifest
    deployment_manifest = f"""
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {name}
  namespace: {namespace}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {name}
  template:
    metadata:
      labels:
        app: {name}
    spec:
      containers:
      - name: website
        image: {image}
        ports:
        - containerPort: {port}
"""
    # Decode the YAML to a dict and create it
    obj = yaml.safe_load(deployment_manifest)
    api.create_namespaced_deployment(namespace, obj)
    kopf.info(body, reason='DeploymentCreated', message=f"Deployment {name} created")

@kopf.on.delete('example.com', 'v1', 'websites')
def delete_website(body, **kwargs):
    """Clean up the Deployment when the Website is deleted."""
    name = body['metadata']['name']
    namespace = body['metadata'].get('namespace', 'default')
    api.delete_namespaced_deployment(name, namespace)
    print(f"Deleted deployment {name}")

Notice how kopf decorators bind your handler to the CR type: @kopf.on.create('example.com', 'v1', 'websites'). The framework passes the body and spec as Python dicts, so you can write familiar Python code.

3. Run it locally (for testing)

Before containerizing, test against minikube locally. Start minikube, then:

# Install kopf and kubernetes client
pip install kopf kubernetes pyyaml

# Run the operator with your kubeconfig
kopf run --verbose operator.py

In another terminal, create a website.yaml:

apiVersion: example.com/v1
kind: Website
metadata:
  name: my-site
spec:
  image: nginx:1.25
  port: 80

Apply it and watch the operator log. You should see something like:

[INFO] Handler 'create_website' succeeded.

And kubectl get deployments shows my-site running.

4. Containerize and deploy

Create a Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY operator.py .
CMD ["kopf", "run", "--verbose", "/app/operator.py"]

Build and push to a registry, then create RBAC and a Deployment for the operator itself. Here’s a minimal deploy-operator.yaml (you’ll need proper ServiceAccount, Role and RoleBinding — see the kopf docs for full YAML):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: website-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: website-operator
  template:
    metadata:
      labels:
        app: website-operator
    spec:
      serviceAccountName: website-operator-sa
      containers:
      - name: operator
        image: yourrepo/website-operator:latest

Once deployed, the operator runs inside the cluster and watches for any new Website CRs.

Compare options / when to choose what

Recent years have seen many ways to write operators. Here’s how they compare:

• kopf shines for Python developers: you already know the language, and you can reuse existing Python libraries for domain logic.
• Operator SDK gives you the most control and performance (Go is native), but requires learning Go and a steeper learning curve.
• Metacontroller lets you write a simple webhook in Python, but you lose some of the convenience of kopf’s decorators.
• Shell/Helm hooks are fine for a one-time job, but they don’t give you the event-driven reconciliation that an operator provides.

Variation to consider: If you only need to react to CR changes and don’t need the full control loop, you can use the Kubernetes client with a simple watch loop. But you’d be reimplementing leader election, retries, and error handling that kopf gives you for free.

Troubleshooting & edge cases

When you build a Python operator, you’ll run into a few common pitfalls. Here’s how to fix them:

• Missing RBAC permissions — You’ll see kubernetes.client.exceptions.ApiException: (403) Forbidden when the operator tries to create a Deployment. Fix: add the appropriate rules to your Role (e.g., deployments: get, list, watch, create, delete).
• CRD schema rejects your YAML — If you try to add a field that’s not defined in the CRD, the API server will refuse. Fix: update the CRD schema first, then apply the CR again.
• Operator crashes on startup — If you’re running locally, load_incluster_config() expects to be inside a cluster. For local testing, use load_kube_config(). A common pattern is to check an env var:

import os
if os.getenv('KUBERNETES_SERVICE_HOST'):
    kubernetes.config.load_incluster_config()
else:
    kubernetes.config.load_kube_config()

• Handlers not triggered — Make sure the resource you’re watching is in the same group/version as the decorator. Check the CRD’s group field matches the string in @kopf.on.create('example.com', 'v1', ...).
• Stale resources after crash — If the operator crashes after creating a Deployment, the Deployment stays. That’s fine — the reconciliation loop will pick up when the operator restarts. But for idempotency, your handlers should check if the resource already exists before creating it.

Pro tip: Always make your handlers idempotent — they should handle the same event twice without creating duplicate objects. Use try/except around create and fall back to patch if it already exists.

What you learned & what's next

You can now explain the core idea behind building a Python operator for custom resources: you define a CRD, write festive Python handlers with kopf, and deploy a controller that reconciles your cluster. You completed a hands-on exercise that creates a Deployment from a custom resource and cleans it up on deletion. That means you’re ready to automate real operational tasks — no more manual kubectl for every new service.

Your next step in the Kubernetes for Python Developers track is to explore advanced operator patterns — such as implementing status subresources, finalizers, and leader election for high availability. Those will make your operators production‑grade. You now have the foundation to go audit existing operators and even contribute to open‑source ones. Keep building!