Manage Secrets with External Secrets Operator

You've mastered deploying workloads and configuring networking, but one invisible threat can undo all that work: leaking secrets. Plain Kubernetes Secrets are base64-encoded, not encrypted, and managing rotation across dozens of microservices is a nightmare. That's where the External Secrets Operator (ESO) steps in — it syncs secrets from external vaults (like AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, HashiCorp Vault) directly into your cluster, so you never have to hardcode sensitive data again.

The problem this lesson solves

Managing secrets directly in Kubernetes is inherently risky. A single commit to a public repo or a misconfigured RBAC rule can expose database passwords, API keys, or TLS certificates. Even with encryption at rest via kube-encrypt, you still face operational pain: rotating a password means updating every Secret YAML, reapplying it, and restarting pods. With External Secrets Operator, you decouple where secrets are stored from how pods consume them, following the principles of GitOps and zero-trust security.

In this lesson, you'll learn how ESO solves these exact pain points: - Centralized secret management — secrets live in a single source of truth (e.g., AWS Secrets Manager). - Automated synchronization — when a secret rotates in your vault, ESO updates the corresponding Kubernetes Secret within seconds. - No plaintext in manifests — your deployments reference a Secret name, never the actual value.

Core concept / mental model

Imagine a bridge between two islands: - Island A: Your external secrets store (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager). - Island B: Your Kubernetes cluster.

The External Secrets Operator is that bridge. It watches for custom resources called ExternalSecret and ClusterExternalSecret. When you apply an ExternalSecret manifest, the operator reaches out to your vault, fetches the secret value, and creates or updates a standard Kubernetes Secret in your namespace.

Key components

• ExternalSecret (namespace-scoped): maps vault secrets to a Kubernetes Secret in the same namespace.
• SecretStore / ClusterSecretStore: defines how to authenticate to the external vault (e.g., IAM role, static token, service principal).
• Provider: the backend service (AWS Secrets Manager, Azure Key Vault, etc.).

Pro tip: Think of SecretStore as a configuration object that tells ESO where to look, and ExternalSecret as the mapping that tells it what to fetch and how to name it locally.

How it works step by step

1. Install the operator in your cluster (via Helm, manifest, or OperatorHub). ESO runs as a Deployment with a controller loop.
2. Define a SecretStore with your vault credentials. For example, an AWS Secrets Manager store might use an IAM role with IRSA (IAM Roles for Service Accounts).
3. Apply an ExternalSecret resource specifying: - The secretStoreRef (which store to use). - The remote key path (e.g., arn:aws:secretsmanager:us-east-1:123456789012:secret:prod-db-abc123). - How to map the remote secret's key/value pairs to the local Secret.
4. Monitor: ESO syncs the secret immediately, then periodically (default refreshInterval: 1h) or on-demand via annotation eso.sync.io/force-sync. Pods read the resulting Secret as usual.

Detailed lifecycle for a rotation

• Your CI/CD updates the secret in AWS Secrets Manager.
• After the refreshInterval, ESO compares the remote secret hash with the local one.
• If changed, ESO patches the local Secret.
• Any pod that mounts that Secret will see the update after the kubelet syncs (usually within 60–90 seconds depending on mount propagation).

Hands-on walkthrough

Prerequisites

• A Kubernetes cluster (local like kind/Minikube or remote).
• kubectl installed.
• Access credentials to an external secrets store. For this example we'll use AWS Secrets Manager with an IAM role attached to the node (for simplicity).

Step 1: Install External Secrets Operator

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace

Verify the operator pod is running:

kubectl get pods -n external-secrets

Step 2: Create a SecretStore pointing to AWS Secrets Manager

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-store
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          accessKeyIDSecretRef:  # optional; if using IRSA, omit
            name: aws-creds
            key: access-key
          secretAccessKeySecretRef:
            name: aws-creds
            key: secret-key

Security note: In production, use IRSA (IAM Roles for Service Accounts) instead of static keys. For a cluster with OIDC provider configured, you can set clusterSecretStore with an IAM role ARN in the pod's service account annotation.

Step 3: Create an ExternalSecret to sync a specific secret

Assume you have a secret in AWS called prod-db-password with a JSON value: {"password":"S3cur3P@ss!"}.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-password
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-store
    kind: SecretStore
  target:
    name: db-secret          # resulting K8s Secret name
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod-db-password
        property: password   # if the remote secret is JSON, pick a key

Apply it:

kubectl apply -f external-secret-db.yaml

Step 4: Verify the K8s Secret was created

kubectl get secret db-secret -o yaml

You'll see a password field. Decode it:

echo "YTM0Y3IzUFAhIQ==" | base64 -d   # example output: S3cur3P@ss!

That's it — your pod can now use db-secret as a volume mount or environment variable.

Compare options / when to choose what

Pro tip: If you already have a vault like AWS Secrets Manager or Azure Key Vault, ESO is the natural fit. If you want a Git-native solution without a vault, Sealed Secrets is simpler.

Troubleshooting & edge cases

Common mistake: SecretStore not applying because of missing namespace / wrong auth

Symptom: ExternalSecret stays Status: Pending Solution: Check the operator logs:

kubectl logs -n external-secrets deploy/external-secrets

Look for errors like access denied or secretRef not found. Ensure your IAM role or static keys have secretsmanager:GetSecretValue permission.

Edge case: The remote secret property only exists after rotation

If your ExternalSecret references a property that doesn't initially exist, ESO will report a sync failure. Use dataFrom with find to pull all keys dynamically:

  dataFrom:
    - extract:
        key: prod-db-password   # extracts all JSON keys

Edge case: SecretStore credentials expire

If you use static credentials (not recommended), rotate them by updating the referenced Secret. Then annotate the SecretStore to force a reload:

kubectl annotate secretstore aws-secrets-store eso.sync.io/force-sync=true

What you learned & what's next

You now understand how to manage secrets with External Secrets Operator — from installing the operator to creating SecretStore and ExternalSecret resources, and syncing sensitive data from an external vault into Kubernetes. You've seen how it automates rotation and separates the secret lifecycle from your application code.

Key takeaways: - ESO bridges external vaults and Kubernetes Secrets. - SecretStore defines where secrets live; ExternalSecret maps them to local Secrets. - Use refreshInterval to control sync frequency. - Prefer IRSA / workload identity over static keys. - Troubleshoot by checking operator logs and SecretStore status.

Next up: Secrets Rotation and Immutable Secrets — learn how to force new pod creation when a secret changes and how to avoid stale secrets in long-running pods.