Kubernetes Node Maintenance: Cordoning, Draining, and Node Management

In Kubernetes clusters, managing nodes often requires excluding them from scheduling to perform maintenance operations. The primary commands for this purpose are cordon, drain, and delete. Each of these commands serves a distinct purpose in controlling pod scheduling and node availability during maintenance.

1. The cordon Command

The cordon command marks a node as unschedulable (SchedulingDisabled). After executing this command, newly created pods will not be scheduled to the node, but existing pods continue running unaffected. This is the least disruptive operation, suitable for scenarios requiring temporary node isolation.

Command usage:

• Prevent scheduling: kubectl cordon <node-name>
• Resume scheduling: kubectl uncordon <node-name>

2. The drain Command

The drain command takes a more comprehensive approach than cordon. It not only marks the node as unschedulable but also systematically evicts pods from the node, migrating them to other available nodes. During this process, drain gracefully terminates pods while respecting PodDisruptionBudgets (PDBs) to ensure application continuity.

The drain method safely evicts pods by waiting for container applications to shut down gracefully before removing the pods. The eviction process works by first deleting pods from the target node and then recreating them on other nodes. To ensure this process occurs without service interruption (achieving "seamless" eviction), it's essential to have more than one pod replica and implement "anti-affinity" strategies to distribute pods across different nodes. In such "multiple pod replicas + anti-affinity" scenarios, the drain process has no impact on container services.

Command usage:

• Evict and prevent scheduling: kubectl drain <node-name> --force --ignore-daemonsets --delete-local-data
• Resume scheduling: kubectl uncordon <node-name>

Important considerations:

• The drain command ignores pods managed by DaemonSets; the --ignore-daemonsets parameter is necessary.
• For pods using local storage, the --delete-local-data parameter may be required to force deletion.

drain is a safe eviction method suitable for pre-maintenance operations like kernel upgrades or hardware replacements. Before implementing drain, ensure pod replicas exceed 1 and anti-affinity strategies are configured to prevent service interruption.

In production environments, Kubernetes pods are typically configured with PodDisruptionBudgets (PDBs) to maintain service availability during pod evictions. Ideally, PDBs should be configured with maxUnavailable set to 0 and maxSurge set to 1. This configuration ensures the total number of pods doesn't decrease during eviction, maintaining service continuity and stability.

replicas: 2
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1          
    maxUnavailable: 0

3. The delete Command

The delete commmand is the most destructive operation, directly removing the node and all its pods from the cluster. This operation is forceful and ignores PDBs and graceful pod termination settings. After using delete, manually restarting kubelet and handling certificate issuance is required to reintegrate the node into the cluster.

Command usage:

• Delete node: kubectl delete node <node-name>

Restoring scheduling (rejoining the Kubernetes cluster):

• After deletion, if the node needs to be rejoined to the Kubernetes cluster, restart the kubelet service on the node. After restart, the node's self-registration feature will allow it to rejoin the cluster and resume schedulable status.
• Additionally: If the system time on the node differs from other nodes after kubelet restart, the node's cluster certificates may become invalid. After kubelet registration, manual approval and TLS certificate signing may be required. For example:

# A node (node02) restarted with system time inconsistent with other nodes
# leads to invalid cluster certificates
[root@k8s-master01 ingress]# kubectl exec -ti test-finhub-app-56df548879-ghlb2 -n wiseco -- /bin/bash
Error from server: error dialing backend: x509: certificate is valid for k8s-master01, not k8s-node02

[root@k8s-master01 ingress]# kubectl get csr
NAME        AGE     REQUESTOR                CONDITION
csr-7zt2w   50m     system:node:k8s-node02  Pending (after node restart)
csr-8sw6k   36m     system:node:k8s-node02  Pending (after node restart)
csr-9jv7z   21m     system:node:k8s-node02  Pending (after node restart)

# Manual approval and TLS certificate signing required
[root@k8s-master01 ingress]# kubectl certificate approve csr-7zt2w
[root@k8s-master01 ingress]# kubectl certificate approve csr-8sw6k
[root@k8s-master01 ingress]# kubectl certificate approve csr-9jv7z

4. Recommended Steps for Smooth Node Maintenance

Typically, when performing smooth maintainance on a node in a Kubernetes cluster—such as upgrades or configuration adjustments—the correct procedure is as follows:

1. Use cordon to temporarily isolate the node from the Kubernetes cluster, marking it as SchedulingDisabled (unschedulable).
2. Use drain to evacuate all pod resources from the node to other available nodes.
3. Perform the maintenance operations on the node, such as upgrades or configuraton adjustments.
4. Use uncordon to restore the node, making it schedulable again in the Kubernetes cluster.

Additionally, to ensure container application services remain uninterrupted during pod eviction with drain, the following conditions must be met:

• The number of pod replicas for services must exceed 1
• "Anti-affinity" strategies must be configured to ensure evicted pods are scheduled to different nodes
• Deployments should use rolling updates with maxUnavailable set to 0 and maxSurge set to 1