When etcd TLS certificates expire, the Kubernetes control plane becomes unstable—API server failures, kubelet connection drops, and cluster-wide unresponsiveness commonly follow. This guide outlines a safe, coordinated procedure to renew etcd’s certificate authority (CA) and member certificates across a multi-node etcd cluster without data loss.

Diagnosis

Begin by verifying symptoms and isolating the root cause:

• Kubelet status: Run systemctl status kubelet; look for connection refusals on port 6443, indicating API server unavailability.
• API server logs: Check container logs (crictl logs $(crictl ps --name kube-apiserver -q)) for repeated failed to store object in database or context deadline exceeded errors—signs of etcd communication breakdown.
• etcd health: Use etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/node-k8s-master1.pem --key=/etc/etcd/ssl/node-k8s-master1-key.pem endpoint health. A failure with x509: certificate has expired or is not yet valid confirms certificate expiration.
• Certificate validity: Inspect expiration dates: ``` openssl x509 -in /etc/etcd/ssl/ca.pem -noout -dates openssl x509 -in /etc/etcd/ssl/member-k8s-master1.pem -noout -dates

Preparation

Before regeneration, ensure redundancy and safety:

1. Locate certificate paths: Confirm location via systemctl cat etcd | grep -E "(cert-file|key-file|trusted-ca-file)".
2. Backup existing certificates on all etcd nodes: ``` mkdir -p /tmp/etcd-certs-backup-$(date +%Y%m%d) mv /etc/etcd/ssl/* /tmp/etcd-certs-backup-$(date +%Y%m%d)/

3. Stop etcd and archive data: ``` systemctl stop etcd tar -czf /tmp/etcd-data-backup-$(date +%Y%m%d).tar.gz -C /var/lib etcd

Certificate Regeneration

Create a standardized OpenSSL configuration supporting SANs for all cluster endpoints:

cat > /etc/etcd/ssl/openssl.cnf <<'EOF'
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ssl_client]
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
subjectAltName = @alt_names

[v3_ca]
basicConstraints = CA:TRUE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
authorityKeyIdentifier = keyid:always,issuer

[alt_names]
DNS.1 = localhost
DNS.2 = etcd.kube-system.svc.cluster.local
DNS.3 = etcd.kube-system.svc
DNS.4 = etcd.kube-system
DNS.5 = etcd
DNS.6 = lb.kubesphere.local
DNS.7 = k8s-master1
DNS.8 = k8s-master2
DNS.9 = k8s-master3
IP.1 = 127.0.0.1
IP.2 = 10.100.1.31
IP.3 = 10.100.1.32
IP.4 = 10.100.1.33
EOF

Generate a new root CA valid for 10 years:

cd /etc/etcd/ssl
openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 3650 -out ca.pem -subj "/CN=etcd-root-ca"

For each node (e.g., k8s-master1, k8s-master2, k8s-master3), generate three distinct certificate types:

• Member cert: For inter-etcd peer communication.
• Admin cert: For administrative CLI access (e.g., etcdctl).
• Node cert: For client-facing endpoints (e.g., API server connections).

Example for k8s-master1:

# Member
openssl genrsa -out member-k8s-master1-key.pem 2048
openssl req -new -key member-k8s-master1-key.pem -out member-k8s-master1.csr -subj "/CN=etcd-member-k8s-master1" -config openssl.cnf
openssl x509 -req -in member-k8s-master1.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out member-k8s-master1.pem -days 3650 -extensions ssl_client -extfile openssl.cnf

# Admin
openssl genrsa -out admin-k8s-master1-key.pem 2048
openssl req -new -key admin-k8s-master1-key.pem -out admin-k8s-master1.csr -subj "/CN=etcd-admin-k8s-master1"
openssl x509 -req -in admin-k8s-master1.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out admin-k8s-master1.pem -days 3650 -extensions ssl_client -extfile openssl.cnf

# Node
openssl genrsa -out node-k8s-master1-key.pem 2048
openssl req -new -key node-k8s-master1-key.pem -out node-k8s-master1.csr -subj "/CN=etcd-node-k8s-master1"
openssl x509 -req -in node-k8s-master1.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out node-k8s-master1.pem -days 3650 -extensions ssl_client -extfile openssl.cnf

Repeat for remaining nodes, then distribute certs using scp or configuration management.

Validation and Deployment

Verify expiration dates:

openssl x509 -in ca.pem -noout -enddate
openssl x509 -in member-k8s-master1.pem -noout -enddate
openssl x509 -in node-k8s-master1.pem -noout -enddate

Update systemd unit files (/etc/systemd/system/etcd.service) per node to reference new certificate paths and correct host-specific flags. For k8s-master1, adjust --name, --initial-advertise-peer-urls, --listen-peer-urls, and --advertise-client-urls accordingly.

Finally, restart etcd simultaneously across all members** to avoid quorum disruption:

# On all nodes
systemctl daemon-reload
systemctl restart etcd

Confirm cluster stability with:

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ssl/ca.pem \
  --cert=/etc/etcd/ssl/node-k8s-master1.pem \
  --key=/etc/etcd/ssl/node-k8s-master1-key.pem \
  endpoint status --write-out=table