etcd serves as the cirtical distributed key-value store for Kubernetes, holding cluster state and configuration data. Regular backups are essential for disaster recovery and cluster migration. This guide outlines a complete procedure for backing up and restoring an etcd cluster using etcdctl, assuming a three-node HA setup deployed via kubeadm.

Prerequisites

• Kubernetes version: 1.23.17
• Three master nodes: master01 (192.168.1.161), master02 (192.168.1.162), master03 (192.168.1.163)
• Access to TLS certificates under /etc/kubernetes/pki/etcd/

Install etcdctl

Download and install etcdctl on one master node:

wget https://gh.monlor.com/https://github.com/etcd-io/etcd/releases/download/v3.4.30/etcd-v3.4.30-linux-amd64.tar.gz
tar -zxf etcd-v3.4.30-linux-amd64.tar.gz
mv etcd-v3.4.30-linux-amd64/etcdctl /usr/local/bin/
chmod +x /usr/local/bin/etcdctl

Copy the binary to other master nodes:

scp /usr/local/bin/etcdctl master02:/usr/local/bin/
scp /usr/local/bin/etcdctl master03:/usr/local/bin/

Configure Environment Variables

Add the following to ~/.bashrc on all nodes:

export ETCDCTL_API=3
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/peer.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/peer.key
export ETCDCTL_ENDPOINTS=192.168.1.161:2379,192.168.1.162:2379,192.168.1.163:2379

Reload the shell configuration:

source ~/.bashrc

Verify Cluster Health

Check endpoint status using a specific endpoint:

ETCDCTL_API=3 \
ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/peer.crt \
ETCDCTL_KEY=/etc/kubernetes/pki/etcd/peer.key \
ETCDCTL_ENDPOINTS=192.168.1.161:2379 \
etcdctl endpoint health --write-out=table

Expected output:

Perform Backup

Create a dedicated backup directory on all nodes:

mkdir -p /opt/etcd_backup

Execute snapshot on a single endpoint (e.g., master01):

ETCDCTL_API=3 \
ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/peer.crt \
ETCDCTL_KEY=/etc/kubernetes/pki/etcd/peer.key \
ETCDCTL_ENDPOINTS=192.168.1.161:2379 \
etcdctl snapshot save /opt/etcd_backup/snap-etcd-$(date +%F-%H-%M-%S).db

Confirm the snapshot file exists:

ls -l /opt/etcd_backup/snap-etcd-*.db

Prepare for Recovery

Delete test resources to simulate data loss:

kubectl delete deployment nginx
kubectl -n ops delete deployment redis-single

Stop etcd on all master nodes by moving the manifest out of the manifests directory:

mv /etc/kubernetes/manifests/etcd.yaml /home/

Backup existing etcd data:

mv /var/lib/etcd /var/lib/etcd-$(date +%F-%H-%M-%S)-backup

Restore Snapshot on Each Node

Transfer the snapshot file to all masters:

scp /opt/etcd_backup/snap-etcd-*.db master02:/opt/etcd_backup/
scp /opt/etcd_backup/snap-etcd-*.db master03:/opt/etcd_backup/

On master01:

ETCDCTL_API=3 etcdctl snapshot restore /opt/etcd_backup/snap-etcd-2024-09-06-11-12-02.db \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --data-dir=/var/lib/etcd \
  --endpoints=https://127.0.0.1:2379 \
  --initial-cluster=master01=https://192.168.1.161:2380,master02=https://192.168.1.162:2380,master03=https://192.168.1.163:2380 \
  --name=master01 \
  --initial-advertise-peer-urls=https://192.168.1.161:2380

On master02:

ETCDCTL_API=3 etcdctl snapshot restore /opt/etcd_backup/snap-etcd-2024-09-06-11-12-02.db \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --data-dir=/var/lib/etcd \
  --endpoints=https://127.0.0.1:2379 \
  --initial-cluster=master01=https://192.168.1.161:2380,master02=https://192.168.1.162:2380,master03=https://192.168.1.163:2380 \
  --name=master02 \
  --initial-advertise-peer-urls=https://192.168.1.162:2380

On master03:

ETCDCTL_API=3 etcdctl snapshot restore /opt/etcd_backup/snap-etcd-2024-09-06-11-12-02.db \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --data-dir=/var/lib/etcd \
  --endpoints=https://127.0.0.1:2379 \
  --initial-cluster=master01=https://192.168.1.161:2380,master02=https://192.168.1.162:2380,master03=https://192.168.1.163:2380 \
  --name=master03 \
  --initial-advertise-peer-urls=https://192.168.1.163:2380

Restart etcd

Restore the etcd manifest on all nodes:

mv /home/etcd.yaml /etc/kubernetes/manifests/

The kubelet will automatically restart etcd pods. Wait 30–60 seconds for the cluster to stabilize.

Validation

Confirm restored resources:

kubectl get deployments -A | grep -E "(nginx|redis-single)"

Verify etcd cluster health:

ETCDCTL_API=3 etcdctl endpoint status --write-out=table

Automate Backup with Script

Create a backup script at /scripts/etcd_backup.sh:

#!/bin/bash

export ETCDCTL_API=3
BACKUP_DIR="/opt/etcd_backup"
SNAPSHOT_FILE="snap-etcd-$(date +%F-%H-%M-%S).db"
SNAPSHOT_PATH="$BACKUP_DIR/$SNAPSHOT_FILE"

mkdir -p "$BACKUP_DIR"

ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/peer.crt \
ETCDCTL_KEY=/etc/kubernetes/pki/etcd/peer.key \
ETCDCTL_ENDPOINTS=192.168.1.161:2379 \
etcdctl snapshot save "$SNAPSHOT_PATH"

if [ $? -eq 0 ]; then
  echo "✅ Backup successful: $SNAPSHOT_PATH"
else
  echo "❌ Backup failed"
  exit 1
fi

Make executable and schedule weekly via cron:

chmod +x /scripts/etcd_backup.sh
crontab -e
# Add line: 0 0 * * 3 /scripts/etcd_backup.sh