Deploying a Kubernetes 1.16.4 HA Cluster on CentOS 7.6 with Keepalived

Environment Overview

Hostname OS Version IP Address Components Hardware Role
master01 CentOS 7.6.1810 172.27.34.3 Docker 18.09.9, Flannel v0.11.0, Keepalived v1.3.5 4C4G control plane
master02 CentOS 7.6.1810 172.27.34.4 Docker 18.09.9, Flannel v0.11.0, Keepalived v1.3.5 4C4G control plane
master03 CentOS 7.6.1810 172.27.34.5 Docker 18.09.9, Flannel v0.11.0, Keepalived v1.3.5 4C4G control plane
work01 CentOS 7.6.1810 172.27.34.93 Docker 18.09.9 4C4G worker node
work02 CentOS 7.6.1810 172.27.34.94 Docker 18.09.9 4C4G worker node
work03 CentOS 7.6.1810 172.27.34.95 Docker 18.09.9 4C4G worker node
VIP N/A 172.27.34.130 Managed by Keeepalived N/A floats on control planes
client CentOS 7.6.1810 172.27.34.234 kubectl only 4C4G management client

All control plane and worker nodes run Kubernetes components kubelet, kubeadm, and kubectl at version v1.16.4. The client node has only kubectl installed.

High-Availability Architecture

The setup uses a primary-backup pattern for control plane components managed by kubeadm:

  • API server: Failover handled by Keepalived VIP.
  • Controller Manager & Scheduler: Internal leader election (default --leader-elect=true).
  • etcd: Built-in clustering via kubeadm on three nodes, tolerating a single node failure.

Common Node Preparation

All control plane and worker machines must complete these steps. Firewall and SELinux are assumed disabled, and Aliyun mirrors configured during OS installation.

Host FQDN and Name Resolution

Set the hostname and populate /etc/hosts:

hostnamectl set-hostname master01
cat >> /etc/hosts << EOF
172.27.34.3    master01
172.27.34.4    master02
172.27.34.5    master03
172.27.34.93   work01
172.27.34.94   work02
172.27.34.95   work03
EOF

UUID and MAC Verification

Ensure uniqueness:

cat /sys/class/net/ens160/address
cat /sys/class/dmi/id/product_uuid

Swap Disabling

swapoff -a
sed -i.bak '/swap/s/^/#/' /etc/fstab

Kernel Tunables for Flannel

Flannel reuqires br_netfilter and the bridge iptables settings.

# Ensure module loads at boot
cat > /etc/sysconfig/modules/br_netfilter.modules << 'EOF'
modprobe br_netfilter
EOF
chmod 755 /etc/sysconfig/modules/br_netfilter.modules

# Persist sysctl settings
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf

Kubernetes Yum Repository

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum clean all && yum -y makecache

Password-less SSH (master01 only)

ssh-keygen -t rsa
ssh-copy-id root@172.27.34.4
ssh-copy-id root@172.27.34.5

Installing Docker CE

Perform on all control plane and worker nodes.

# Prerequisites
yum install -y yum-utils device-mapper-persistent-data lvm2

# Add Docker repository
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# Install a specific release
yum install -y docker-ce-18.09.9 docker-ce-cli-18.09.9 containerd.io

# Configure image acceleration and cgroup driver
mkdir -p /etc/docker
cat <<'EOF' > /etc/docker/daemon.json
{
  "registry-mirrors": ["https://v16stybc.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

systemctl daemon-reload
systemctl enable --now docker

Keepalived Configuration

On each control plane node, install Keepalived and define the VRRP instance. The shared VIP is 172.27.34.130.

master01 (priority 100):

! Configuration File for keepalived
global_defs {
   router_id master01
}
vrrp_instance VI_1 {
    state MASTER
    interface ens160
    virtual_router_id 50
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.27.34.130
    }
}

master02 (priority 90) and master03 (priority 80) use state BACKUP. After placing the configs, start and enable the service:

yum install -y keepalived
service keepalived start
systemctl enable keepalived

Kubernetes Component Installation

Install on every cluster node:

yum install -y kubelet-1.16.4 kubeadm-1.16.4 kubectl-1.16.4
systemctl enable kubelet && systemctl start kubelet

Pull required images from an Alibaba Cloud mirror to avoid network restrictions:

#!/bin/bash
ALI_REPO=registry.cn-hangzhou.aliyuncs.com/loong576
KUBE_VER=v1.16.4
IMAGES=($(kubeadm config images list --kubernetes-version=$KUBE_VER | awk -F '/' '{print $NF}'))
for img in ${IMAGES[@]}; do
  docker pull $ALI_REPO/$img
  docker tag $ALI_REPO/$img k8s.gcr.io/$img
  docker rmi -f $ALI_REPO/$img
done

Bootstrapping the First Master

Create a configuration manifest kubeadm-config.yaml:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.4
apiServer:
  certSANs:
    - master01
    - master02
    - master03
    - 172.27.34.3
    - 172.27.34.4
    - 172.27.34.5
    - 172.27.34.130
controlPlaneEndpoint: "172.27.34.130:6443"
networking:
  podSubnet: "10.244.0.0/16"

Initialize the cluster and export the admin credentials:

kubeadm init --config=kubeadm-config.yaml
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source ~/.bash_profile

Deploy Flannel overlay:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

Adding Remaining Control Plane Nodes

Distribute certificates from master01:

declare -a TARGETS=("172.27.34.4" "172.27.34.5")
for host in ${TARGETS[@]}; do
  ssh $host 'mkdir -p /etc/kubernetes/pki/etcd'
  scp /etc/kubernetes/pki/{ca.crt,ca.key,sa.key,sa.pub,front-proxy-ca.crt,front-proxy-ca.key} $host:/etc/kubernetes/pki/
  scp /etc/kubernetes/pki/etcd/{ca.crt,ca.key} $host:/etc/kubernetes/pki/etcd/
done

On master02 and master03, execute the join command obtained during initialization, appending --control-plane:

kubeadm join 172.27.34.130:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> --control-plane

To allow kubectl usage from those nodes, copy the admin config:

scp master01:/etc/kubernetes/admin.conf /etc/kubernetes/
export KUBECONFIG=/etc/kubernetes/admin.conf

Registering Worker Nodes

Run the standard join command on each worker:

kubeadm join 172.27.34.130:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Confirm the cluster:

kubectl get nodes

Client Machine Setup

Install kubectl and obtain the cluster configuration:

yum install -y kubectl-1.16.4
mkdir -p /etc/kubernetes
scp 172.27.34.3:/etc/kubernetes/admin.conf /etc/kubernetes/
echo 'export KUBECONFIG=/etc/kubernetes/admin.conf' >> ~/.bash_profile
source ~/.bash_profile

Dashboard Deployment

From the client node, fetch and customize the manifest:

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

# Adjust image repository
sed -i 's/kubernetesui/registry.cn-hangzhou.aliyuncs.com\/loong576/g' recommended.yaml

# Expose via NodePort
sed -i '/targetPort: 8443/a\      nodePort: 30001\n\  type: NodePort' recommended.yaml

Create an admin ServiceAccount and ClusterRoleBinding at the end of the same file:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-admin
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dashboard-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: dashboard-admin
  namespace: kubernetes-dashboard

Apply and retrieve the login token:

kubectl apply -f recommended.yaml
kubectl describe secret $(kubectl get secret -n kubernetes-dashboard | grep dashboard-admin-token | awk '{print $1}') -n kubernetes-dashboard

Access the dashboard at https://VIP:30001 using the displayed token.

High-Availability Validation

  • Initially, the VIP, controller-manager, and scheduler leaders reside on master01.
  • After shutting down master01 (init 0): the VIP shifts to master02, APIs remain reachable, and pods can be created.
  • When master02 is also stopped, the VIP moves to master03. However, with two out of three etcd nodes offline, the cluster loses quorum, causing API requests to time out, illustrating the limits of a three-node HA design.

Tags: kubernetes High Availability v1.16.4 Keepalived centos

Posted on Mon, 24 Aug 2026 16:08:17 +0000 by Saethyr

Hot Tags

©2026 Freaks City