Managing Kubernetes Secrets

You've got your Python service containerized, Deployments rolling, and Services routing traffic — but your database password is still sitting in plain text inside your YAML, or worse, hardcoded in your image. That's a security incident waiting to happen. In this lesson, you'll learn how to manage Kubernetes secrets with kubectl — the hands-on way to store sensitive data like API keys, tokens, and passwords securely in Kubernetes, without baking them into your images or committing them to git.

The problem this lesson solves

ConfigMaps are perfect for non-sensitive configuration like feature flags or URLs. But the moment you need to store a database password, a Stripe API key, or a JWT signing secret, ConfigMaps are the wrong tool. Secrets in plain text can be read by anyone with access to your cluster, get committed to version control, or leaked through logs. The core problem is simple: how do you store and inject sensitive data into your Python pods without exposing it in your manifests or your source code?

Kubernetes answers this with Secrets — a built-in API object designed to hold small amounts of sensitive data like usernames, passwords, and tokens. Instead of hardcoding credentials in your Python code, you store them as a Secret and mount them as environment variables or files inside your pods. This separation keeps your code clean and your credentials out of your repositories.

By the end of this lesson, you'll be able to create, inspect, and use Secrets from the command line with kubectl, and you'll understand best practices for keeping them safe in production.

Core concept / mental model

Think of a Secret as a secure envelope for your configuration. Unlike a ConfigMap, which stores plain-text key-value pairs, a Secret stores your data base64-encoded. That's not encryption — it's just encoding — but it adds a small layer of obscurity and is the standard format Kubernetes expects. The real security comes from how you manage access to the Secret and how you control who can read it (via RBAC) and whether it's encrypted at rest (via your cluster's configuration).

Here's the mental model:

• Secret = a Kubernetes API object that holds one or more key-value pairs.
• Key = the name your app uses (e.g., DB_PASSWORD).
• Value = the sensitive data, base64-encoded in the manifest.
• Injection = you reference the Secret in a Pod spec, and Kubernetes exposes it as an environment variable or a file.

Your Python code doesn't know the difference between a ConfigMap and a Secret — it just reads an env var or a file. The magic happens before your container starts.

How it works step by step

1. Create the Secret — You can create it imperatively with kubectl create secret, from a file, or from a YAML manifest. Each method scales differently for production.

2. Refer to the Secret — In your Pod or Deployment spec, you add an env entry with valueFrom.secretKeyRef (for env vars) or a volume mount with secret type (for files).

3. Inject — Kubernetes reads the Secret, decodes the value, and injects it into the container at runtime.

4. Read it in your Python code — If you used an env var, just read os.environ. If you mounted a file, read the file from the mounted path.

5. Update and rotate — You can update a Secret and restart your pods (or use a tool like reloader or the kubectl rollout restart command) to pick up the change.

Hands-on walkthrough

Let's get your hands dirty. We'll create a Secret for a hypothetical PostgreSQL database that your Python app uses.

Step 1: Create a Secret imperatively

Open your terminal, make sure you're connected to a cluster (e.g., minikube or kind), and run:

# Create a Secret with two key-value pairs
kubectl create secret generic db-credentials \
  --from-literal=DB_USER=admin \
  --from-literal=DB_PASSWORD=S3cretP@ss

Pro tip: The --from-literal syntax is great for quick tests, but for anything serious, use --from-file to avoid putting secrets in your shell history.

You can also create a Secret from a file. Create a file named db-password.txt with your password, then:

kubectl create secret generic db-password \
  --from-file=DB_PASSWORD=./db-password.txt

Step 2: Inspect the Secret

# List all secrets in the default namespace
kubectl get secrets

# Show details (note that values are base64-encoded)
kubectl describe secret db-credentials

# Get the raw YAML (the data is base64-encoded)
kubectl get secret db-credentials -o yaml

Expected output snippet:

apiVersion: v1
data:
  DB_PASSWORD: UzNjcmlwdEBQc3M=
  DB_USER: YWRtaW4=
kind: Secret
metadata:
  name: db-credentials
  namespace: default

Pro tip: kubectl get secret db-credentials -o yaml shows the base64 representation. To see the actual value safely, use kubectl get secret db-credentials -o jsonpath='{.data.DB_PASSWORD}' | base64 --decode — but only for troubleshooting, never in production logs.

Step 3: Create a Deployment that uses the Secret

Save this as app-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: python-app
  template:
    metadata:
      labels:
        app: python-app
    spec:
      containers:
      - name: app
        image: python:3.11-slim
        command: ["sh", "-c", "echo $DB_USER && sleep 3600"]
        env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: DB_USER
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: DB_PASSWORD

Apply it:

kubectl apply -f app-deployment.yaml

Step 4: Verify the env vars in your pod

# Get the pod name
kubectl get pods

# Exec into the pod and echo the env vars
kubectl exec -it <pod-name> -- sh
# Inside the pod:
echo $DB_USER
echo $DB_PASSWORD

You'll see admin and S3cretP@ss printed. Your Python app inside the container can now read them from os.environ.

Step 5: Mount a Secret as a file

Sometimes your Python app expects credentials in a file (e.g., a service account JSON). Here's how:

spec:
  containers:
  - name: app
    image: python:3.11-slim
    volumeMounts:
    - name: creds
      mountPath: "/etc/creds"
      readOnly: true
  volumes:
  - name: creds
    secret:
      secretName: db-credentials

After applying, exec in and run cat /etc/creds/DB_USER — you'll see the value. The key becomes the filename.

Now, in Python, reading a secret mounted as a file

import os

db_user = os.environ['DB_USER']  # Env var approach
db_password = os.environ['DB_PASSWORD']

# File mount approach
with open('/etc/creds/DB_USER', 'r') as f:
    file_user = f.read().strip()

print(f"Database user: {db_user}")  # Output: Database user: admin
print(f"File-based user: {file_user}")

Compare options / when to choose what

When to use what:

• Local mini-cluster / learning: Use kubectl create secret with --from-literal for speed.
• Dev environment without strict security: Use --from-file to keep secrets out of your shell history, but still in the cluster.
• Production with strict controls: Use a tool like External Secrets Operator or Vault to sync secrets from a secure store, plus encryption at rest on your cluster.

Pro tip: Never store production secrets in a YAML file that lives in git. Instead, generate manifests from a CI pipeline that sources from a secure vault.

Troubleshooting & edge cases

Secret appears empty in pod

• Symptom: echo $DB_PASSWORD returns nothing.
• Cause: The secret key doesn't match, or the pod started before the Secret existed.
• Fix: Check the spelling in secretKeyRef. Then re-create the Secret and kubectl rollout restart deployment/python-app.

Base64 decoding issues

• Symptom: Your value has a trailing newline or looks garbled.
• Cause: --from-file includes the file's newline by default.
• Fix: Use echo -n when creating from a literal, or strip it in Python with .strip().

Secret not updated in running pods

• Symptom: You changed the Secret but the pod still has the old value.
• Cause: Env vars are set at container start; Kubernetes does not automatically restart pods when a Secret changes.
• Fix: Use kubectl rollout restart deployment/<name> so all replicas re-read the Secret.

Permission denied reading Secret

• Symptom: kubectl get secret returns 403.
• Cause: RBAC restrictions.
• Fix: Have your cluster admin grant your service account the get and list permissions on secrets.

What you learned & what's next

In this lesson, you learned how to manage Kubernetes secrets with kubectl — create secrets imperatively, inspect them, and inject them into pods via env vars or file mounts. You now understand the mental model of Secrets, the trade-offs between creation methods, and how to troubleshoot common issues. Your Python apps can now read sensitive configuration without exposing it in your codebase.

Next, you'll dive into persistent volumes to give your Python services durable storage that survives pod restarts. After that, you'll explore Helm charts to package and deploy your apps with a single command. Those skills will let you take your Python services from a single pod to a fully automated, production-grade deployment.