Back Up and Restore etcd Data

Your Kubernetes cluster is a black box until etcd, the brain storing every pod, service, and config, goes silent. A single corrupt snapshot or misconfigured backup can mean total data loss — hours of re-creation, broken state, and lost debugging context. In production, etcd is the most critical yet often most neglected component; mastering backup and restore gives you the power to survive disk failure, accidental deletion, or a misapplied manifest.

The problem this lesson solves

etcd is the only source of truth for your cluster's state. Every Pod, ConfigMap, Secret, Deployment, and RBAC binding lives inside its key-value store. If etcd crashes and you lack a recent backup, you cannot simply "restart" — you must rebuild the entire cluster from scratch, losing all workload history and persistent metadata. The pain grows exponentially with multi-master setups: a split-brain scenario or a snapshot that silently decays overnight can leave operators scrambling to restore from outdated snapshots. This lesson eliminates that fear by showing you exactly how to create, verify, and restore etcd snapshots.

Core concept / mental model

Think of etcd as the brain of Kubernetes — it stores every object's current state. A backup is a frozen brain scan: a single snapshot file (.db extension) that captures all keys and values at one point in time. Restoring is like replaying that scan into a fresh brain, but only if the snapshot was taken when the brain was healthy.

Pro Tip: A snapshot is not just a copy of the etcd data directory. It is a point-in-time, consistent snapshot that etcd itself produces via etcdctl snapshot save. Do not attempt to copy /var/lib/etcd directly while etcd is running — you will get a corrupted backup.

Key definitions

• Snapshot: A single file containing all etcd data at a specific moment.
• etcdctl: The CLI client that interacts with etcd — the only tool you should use for backup and restore.
• Member: Each etcd node in the cluster. In production, you typically have 3 or 5 nodes (odd number).
• snapshot status: Command to verify a snapshot file is valid.

How it works step by step

Backup and restore follows a simple but strict sequence. Here is the mental flow:

1. Locate etcd — On master nodes, etcd runs as a static pod or systemd service. Find its endpoint (usually https://127.0.0.1:2379).
2. Take a snapshot — Using etcdctl (v3 API) with the correct TLS certificates (--cacert, --cert, --key).
3. Verify the snapshot — Run etcdctl snapshot status to confirm the file is not corrupt.
4. Backup the snapshot — Store it off-cluster: Stash, S3, or even a second disk.
5. Restore — Stop etcd (if running), clear the data directory, then run etcdctl snapshot restore to create a new member with the snapshot's data.
6. Start etcd — Restart the etcd service or static pod. The cluster re-joins and sees the restored state.

Pro Tip: Always test your restore procedure in a staging cluster first. A backup you have never restored is a prayer, not a plan.

Hands-on walkthrough

We will simulate a single-master cluster (or a multi-master with one node). These commands assume you have etcdctl installed and access to the etcd certificates on a control plane node. If you use a managed Kubernetes (EKS, GKE, AKS), you do not have direct etcd access — use the provider's backup plan instead.

1. Check etcd endpoint and version

# List etcd pods (if static)
kubectl get pods -n kube-system | grep etcd

# SSH into the control plane node and check etcdctl version
etcdctl version

2. Take a snapshot

# Use the certs from /etc/kubernetes/pki/etcd
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot-$(date +%Y%m%d%H%M).db

Expected output:

Snapshot saved at /backup/etcd-snapshot-202503151200.db

3. Verify the snapshot

ETCDCTL_API=3 etcdctl --write-out=table snapshot status /backup/etcd-snapshot-202503151200.db

Expected output:

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| abcdef12 |   123456 |       1024 |      12 MB |
+----------+----------+------------+------------+

A hash that is not 0000000000000000 means the snapshot is valid.

4. Simulate a disaster — restore from snapshot

First, stop etcd:

# If static pod
mv /etc/kubernetes/manifests/etcd.yaml /tmp/
# Wait for pod to stop
kubectl get pods -n kube-system | grep etcd

Then restore:

# Remove the old data directory
rm -rf /var/lib/etcd

# Restore to a new data directory
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-202503151200.db \
  --data-dir=/var/lib/etcd-restore \
  --name=master-node \
  --initial-cluster=master-node=https://127.0.0.1:2380 \
  --initial-cluster-token=etcd-cluster

Expected output: (no output means success)

Move the restored data into place:

mv /var/lib/etcd-restore /var/lib/etcd

Restart etcd:

mv /tmp/etcd.yaml /etc/kubernetes/manifests/
# Wait for etcd pod to come back
kubectl get pods -n kube-system -w

Verify cluster health:

kubectl cluster-info
kubectl get nodes

All your namespaces, deployments, and configs should reappear.

5. Automate the backup (optional but recommended)

Create a cron job on the control plane node:

# /etc/cron.d/etcd-backup
0 */6 * * * root /usr/local/bin/etcd-backup.sh

Where etcd-backup.sh contains the etcdctl snapshot save command and copies the file to S3 via aws s3 cp.

Pro Tip: For multi-master clusters, take the snapshot from one member only — you do not need to back up every node. But keep snapshots from all nodes in case one is corrupted.

Compare options / when to choose what

When to choose what:

• Self-managed cluster (kubeadm, on-premises): Use etcdctl snapshot save with a cron job and off-cluster storage.
• Managed Kubernetes: Use the provider's backup feature (EKS backup via AWS Backup, GKE backup via Snapshots).
• High-volume clusters (>10k keys): Consider periodic incremental snapshots (via etcd v3.5+ snapshot retention) and separate S3 lifecycle policies.

Troubleshooting & edge cases

Snapshot file is empty or hash is zeroes

Cause: The endpoint was unreachable or the snapshot was interrupted.

Fix: Run etcdctl endpoint health first, then retry with --debug flag.

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 endpoint health

Restored cluster has wrong node names

Cause: The --name and --initial-cluster flags do not match the etcd configuration.

Fix: Extract the original member name from the snapshot or use --skip-force-new-cluster only if you understand the risks. For production, always specify --name correctly.

Cluster fails to start after restore

Symptoms: etcd pod crash-looping, kubectl get nodes returns nothing.

Checklist:

1. Verify snapshot was taken from the same cluster (not a different one).
2. Ensure the --data-dir path matches the --data-dir in the etcd manifest.
3. Confirm TLS certificates are still valid — restore does not change certificates.
4. Check etcd logs: journalctl -u etcd -f or kubectl logs <etcd-pod>.

Restoring to a fresh node (scale-up scenario)

When adding a new etcd member, you do not restore a snapshot. Instead, join the new member to the existing cluster using etcdctl member add. Snapshot restore is only for disaster recovery or when replacing a dead member completely.

What you learned & what's next

You now know how to back up and restore etcd data — the brain of your Kubernetes cluster. You can take a snapshot, verify its integrity, and restore it to a clean state. You understand that etcdctl snapshot save and etcdctl snapshot restore are your go-to commands, and that testing the restore process is as critical as the backup itself.

Next lesson: Now that your cluster's state is safe, learn how to update and roll back a deployment — because even with backups, you want to recover quickly from faulty rollouts.