Kubernetes High-Availability Binary Deployment Guide v1.23.x

High-Availability Architecture
Kubernetes Cluster Components
- etcd
- kube-apiserver
- kube-scheduler
- kube-controller-manager
- kubelet
- kube-proxy
- kubectl
- High-Availability Analysis
- Load Balancer Node Design
Environment Preparation
- Environment Planning
- Host Resolution Configuration
- Required Tools Installation
- System Services Configuraton
- SSH Key Setup
Core Components Installation
- Containerd Runtime Configuration
- Docker Runtime Configuration
- Kubernetes and etcd Installation
Certificate Generation
- Certificate Tools Setup
- etcd Certificates
- Kubernetes Component Certificates
High-Availability Configuration
- HA Proxy and Keepalived Installation
- HA Proxy Configuration
- Keepalived Configuration
- Service Startup
- Health Cheeck Script
Kubernetes Component Configuration
- etcd Configuration
- API Server Configuration
- Controller Manager Configuration
- Scheduler Configuration
TLS Bootstrapping Configuration
Worker Node Configuration
Calico Network Plugin Installation
CoreDNS Installation
Metrics Server Installation
Dashboard Installation
Cluster Verification

Kubernetes Cluster Components

Kubernetes operates on a master-worker architecture where master nodes handle core scheduling, management, and orchestration, while worker nodes execute user workloads. In Kubernetes terminology, master nodes are often referred to as Control Plane nodes, and worker nodes are simply called nodes.

Tip: Master nodes typically run API Server, Scheduler, and Controller Manager components, while worker nodes run Kubelet and Kube-Proxy.

The Control Plane represents the core control infrastructure of the entire cluster, essentially an enhanced version of master processes. In Kubernetes, the Control Plane typically runs on master nodes. By default, master nodes do not run application workloads—all workloads are delegated to worker nodes.

etcd

etcd serves as the cluster's primary database, storing the entire cluster state. It handles service discovery and configuration sharing across nodes. As a distributed key-value storage system, etcd maintains cluster state including Pod, Service, and other object information.

Kubernetes cluster etcd databases should have a backup strategy. An alternative HA deployment approach抽离es the etcd database from containers, deploying it separately as a highly available database to provide stable storage for Kubernetes.

kube-apiserver

The API Server provides the sole entry point for resource operations, offering authentication, authorization, access control, API registration, and discovery mechanisms. As the cluster's unified gateway and coordinator among components, it exposes HTTP APIs for all object resource operations—create, read, update, delete, and watch—processing requests before storing them in etcd.

kube-scheduler

The Scheduler performs resource orchestration, placing Pods onto appropriate nodes based on predefined scheduling policies. It manages node resource allocation, receiving Pod creation tasks from the API Server and selecting optimal nodes using sophisticated scheduling algorithms.

kube-controller-manager

The Controller Manager maintains cluster state through background operations including failure detection, automatic scaling, and rolling updates. It manages system controllers that handle node status, Pod counts, and Pod-Service relationships. Each resource type has a corresponding controller managed by the Controller Manager.

Key controllers include:

Node Controller: Handles notifications and responses when node failures occur
Replication Controller: Maintains correct Pod replica counts across the system
Endpoints Controller: Populates endpoint objects linking Services to Pods
Service Account & Token Controllers: Creates default accounts and API access tokens for new namespaces

kubelet

Kubelet manages container lifecycles on each worker node, handling Pod specifications, container images, volumes, and network/storage plugins. As the agent representing the master on worker nodes, it receives Pod allocation tasks, manages containers, periodically retrieves container status, and reports to the API Server. Kubelet converts Pod specifications into running containers, handling lifecycle events, volume mounting, secret retrieval, and status reporting.

kube-proxy

Kube-Proxy enables service discovery and load balancing within the cluster. Running on each worker node, it acts as a network proxy for Pods, retrieving service information from etcd and implementing appropriate proxy strategies. It maintains network rules and handles Layer 4 load balancing operations.

Due to performance considerations, many production deployments replace Kube-Proxy with Ingress Controllers integrating HAProxy or Nginx for service proxying.

kubectl

kubectl is the command-line client tool that formats user commands and sends them to the API Server, serving as the primary interface for cluster operations.

High-Availability Analysis

All API calls from the cluster terminate at the API Server, which communicates directly with etcd. A single API Server instance creates a single point of failure—shutdown or crash prevents stopping, updating, or creating new Pods, Services, or controllers. Similarly, etcd data loss prevents API Server startup.

Key HA requirements include:

Cluster State Persistence: Kubernetes cluster state is stored in etcd, which provides reliability through distribution across multiple nodes. etcd clusters require at least three nodes with an odd number to prevent network splits.
API Server Redundancy: Kubernetes API Servers are stateless, retrieving all necessary data from etcd. Multiple API Servers can run simultaneously without coordination, with a load balancer providing transparent access for users and worker nodes.
Master Election: Certain master components (Scheduler and Controller Manager) cannot run multiple active instances simultaneously. Running these in leader election mode ensures only one instance is active at a time, with standby instances ready to assume leadership upon failure.
Kubernetes HA: Pod and Service high availability is automatically ensured by Kubernetes when critical cluster components maintain high availability.

Load Balancer Node Design

Load balancer nodes facilitate communication between worker nodes and the master cluster. Since load balancers operate outside Kubernetes and are not monitored by the Controller Manager, load balancer failures isolate nodes from masters. Keepalived provides high availability for load balencers by monitoring services and migrating virtual IPs (VIPs) to backup nodes during failures.

Environment Preparation

Environment Planning

Host Configuration

Hostname	IP Address	Role
master01	10.0.0.171	Master Node
master02	10.0.0.172	Master Node
master03	10.0.0.173	Master Node
node01	10.0.0.174	Worker Node
VIP	10.0.0.200	Keepalived Virtual IP

Network Configuration and Version Information

Configuration	Value
Operating System	CentOS 7.9
Docker Version	docker-ce-20.10.x
Pod CIDR	172.16.0.0/16
Service CIDR	10.96.0.0/16

Host Resolution Configuration

cat >> /etc/hosts <<'EOF'
10.0.0.171 master01
10.0.0.172 master02
10.0.0.173 master03
10.0.0.174 node01
EOF

Required Tools Installation

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git -y

System Services Configuration

# Disable firewall, SELinux, dnsmasq, and NetworkManager
systemctl disable --now firewalld
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

# Verify SELinux configuration
grep ^SELINUX= /etc/selinux/config

# Disable swap partition
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

# Install and configure NTP time synchronization
rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install ntpdate -y

# Sync time on all nodes
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com

# Add to crontab
crontab -e
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com

# Configure system limits
ulimit -SHn 65535
cat >> /etc/security/limits.conf <<'EOF'
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

SSH Key Setup for Master01

# Generate SSH key and distribute to all nodes
ssh-keygen
for host in master02 master03 node01; do
    ssh-copy-id -i .ssh/id_rsa.pub $host
done

# Clone installation repository
cd /root/
git clone https://github.com/dotbalo/k8s-ha-install.git
# Alternative: https://gitee.com/dukuan/k8s-ha-install.git
scp -r k8s-ha-install/ 10.0.0.161:~

# Verify cloned repository
ls -l k8s-ha-install/

# Update all systems (excluding kernel) and reboot
yum update -y --exclude=kernel* && reboot

# Upgrade CentOS 7 kernel to 4.19
cd /root/
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm

# Distribute to other master nodes
for host in master02 master03; do
    scp kernel-ml-*.rpm $host:/root/
done

# Install kernel packages on all nodes
cd /root/
yum localinstall -y kernel-ml-*

# Set kernel boot order
grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby-default-kernel)"

# Verify default kernel
grubby --default-kernel

# Reboot all nodes and verify kernel version
reboot
uname -r

# Install IPVS and related tools on all nodes
yum install ipvsadm ipset sysstat libnetfilter_conntrack-devel -y

# Configure IPVS modules on all nodes
cat > /etc/modules-load.d/ipvs.conf <<'EOF'
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF
systemctl enable --now systemd-modules-load.service

# Configure Kubernetes required kernel parameters on all nodes
cat <<eof> /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
net.ipv4.conf.all.route_localnet = 1
vm.overcommit_memory = 1
vm.panic_on_oom = 0
fs.inotify.max_user_watches = 89100
fs.file-max = 52706963
fs.nr_open = 52706963
net.netfilter.nf_conntrack_max = 2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF

sysctl --system

# Reboot to ensure kernel parameters persist
reboot
lsmod | grep -e ip_vs -e nf_conntrack
</eof>

Core Components Installation

Containerd Runtime Configuration

# Configure container repositories
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo

# Install containerd on all nodes
yum install docker-ce-20.10.* docker-ce-cli-20.10.* containerd -y

# Configure required modules for containerd
cat <<eof br_netfilter="" eof="" load="" modprobe="" modules="" overlay="" sudo="" tee=""></eof>

# Configure containerd kernel parameters
cat <<eof eof="" net.bridge.bridge-nf-call-ip6tables="1" net.bridge.bridge-nf-call-iptables="1" net.ipv4.ip_forward="1" sudo="" sysctl="" tee=""></eof>

# Generate containerd configuration
mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml

# Configure SystemdCgroup in containerd runtime options
vim /etc/containerd/config.toml
# Find containerd.runtimes.runc.options and add: SystemdCgroup = true

# Update sandbox image registry
sed -i 's|k8s.gcr.io/pause:|registry.cn-hangzhou.aliyuncs.com/google_containers/pause:|g' /etc/containerd/config.toml

# Start and enable containerd
systemctl daemon-reload
systemctl enable --now containerd

# Configure crictl
cat > /etc/crictl.yaml <<eof debug:="" eof="" false="" image-endpoint:="" runtime-endpoint:="" timeout:="" unix:=""></eof>

Docker Runtime Configuration

# Install Docker components
yum install docker-ce-20.10.* docker-ce-cli-20.10.* -y

# Configure Docker with systemd cgroup driver
mkdir -pv /etc/docker
cat <<eof daemon-reload="" docker="" enable="" eof="" start="" status="" sudo="" systemctl="" tee=""></eof>

Kubernetes and etcd Installation

# Download Kubernetes binaries
wget https://dl.k8s.io/v1.23.0/kubernetes-server-linux-amd64.tar.gz --no-check-certificate

# Extract Kubernetes components
tar -xf kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin \
    kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

# Extract etcd
tar -zxvf etcd-v3.5.1-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin \
    etcd-v3.5.1-linux-amd64/etcd{,ctl}

# Verify installations
kube-apiserver --version
etcdctl version

# Distribute components to all nodes
master_hosts='master02 master03'
worker_hosts='node01'

# Send to master nodes
for node in $master_hosts; do
    scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $node:/usr/local/bin/
    scp /usr/local/bin/etcd* $node:/usr/local/bin/
done

# Send to worker nodes
for node in $worker_hosts; do
    scp /usr/local/bin/kube{let,-proxy} $node:/usr/local/bin/
done

# Create CNI directory on all nodes
mkdir -p /opt/cni/bin

# Switch to correct branch
cd /root/k8s-ha-install && git checkout manual-installation-v1.23.x

Certificate Generation

Certificate Tools Setup

# Download CFSSL tools
wget "https://pkg.cfssl.org/R1.2/cfssl_linux-amd64" -O /usr/local/bin/cfssl --no-check-certificate
wget "https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson --no-check-certificate
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

etcd Certificates

# Create certificate directories on all master nodes
mkdir /etc/etcd/ssl -p

# Create Kubernetes PKI directory on all nodes
mkdir -p /etc/kubernetes/pki

# Generate etcd CA certificate
cd /root/k8s-ha-install/pki
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca

# Generate etcd server certificate
cfssl gencert \
   -ca=/etc/etcd/ssl/etcd-ca.pem \
   -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
   -config=ca-config.json \
   -hostname=127.0.0.1,master01,master02,master03,10.0.0.171,10.0.0.172,10.0.0.173 \
   -profile=kubernetes \
   etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd

# Distribute certificates to other master nodes
master_hosts='master02 master03'
for node in $master_hosts; do
    ssh $node "mkdir -p /etc/etcd/ssl"
    for cert_file in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem; do
        scp /etc/etcd/ssl/${cert_file} $node:/etc/etcd/ssl/${cert_file}
    done
done

Kubernetes Component Certificates

# Generate Kubernetes CA certificate
cd /root/k8s-ha-install/pki
cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca

# Generate API Server certificate
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -hostname=10.96.0.1,10.0.0.200,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,10.0.0.171,10.0.0.172,10.0.0.173 \
   -profile=kubernetes \
   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

# Generate front proxy certificates
cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca
cfssl gencert \
   -ca=/etc/kubernetes/pki/front-proxy-ca.pem \
   -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client

# Generate Controller Manager certificate and kubeconfig
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager

kubectl config set-cluster kubernetes \
   --certificate-authority=/etc/kubernetes/pki/ca.pem \
   --embed-certs=true \
   --server=https://10.0.0.200:16443 \
   --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager@kubernetes \
   --cluster=kubernetes \
   --user=system:kube-controller-manager \
   --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
   --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
   --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
   --embed-certs=true \
   --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager@kubernetes \
   --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# Generate Scheduler certificate and kubeconfig
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler

kubectl config set-cluster kubernetes \
   --certificate-authority=/etc/kubernetes/pki/ca.pem \
   --embed-certs=true \
   --server=https://10.0.0.200:16443 \
   --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
   --client-certificate=/etc/kubernetes/pki/scheduler.pem \
   --client-key=/etc/kubernetes/pki/scheduler-key.pem \
   --embed-certs=true \
   --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config set-context system:kube-scheduler@kubernetes \
   --cluster=kubernetes \
   --user=system:kube-scheduler \
   --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config use-context system:kube-scheduler@kubernetes \
   --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# Generate Admin certificate and kubeconfig
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin

kubectl config set-cluster kubernetes \
   --certificate-authority=/etc/kubernetes/pki/ca.pem \
   --embed-certs=true \
   --server=https://10.0.0.200:16443 \
   --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config set-credentials kubernetes-admin \
   --client-certificate=/etc/kubernetes/pki/admin.pem \
   --client-key=/etc/kubernetes/pki/admin-key.pem \
   --embed-certs=true \
   --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config set-context kubernetes-admin@kubernetes \
   --cluster=kubernetes \
   --user=kubernetes-admin \
   --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config use-context kubernetes-admin@kubernetes \
   --kubeconfig=/etc/kubernetes/admin.kubeconfig

# Generate ServiceAccount key pair
openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

# Distribute Kubernetes certificates to other master nodes
for node in master02 master03; do
    for cert in $(ls /etc/kubernetes/pki | grep -v etcd); do
        scp /etc/kubernetes/pki/${cert} $node:/etc/kubernetes/pki/${cert}
    done
    for config in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do
        scp /etc/kubernetes/${config} $node:/etc/kubernetes/${config}
    done
done

High-Availability Configuration

HA Proxy and Keepalived Installation

# Install HA components on all master nodes
yum install keepalived haproxy -y

HA Proxy Configuration

# Create HA Proxy configuration
mkdir /etc/haproxy/
cat > /etc/haproxy/haproxy.cfg <<eof backend="" balance="" bind="" check="" client="" connect="" default-server="" default_backend="" defaults="" downinter="" eof="" err="" fall="" frontend="" global="" http="" http-keep-alive="" http-request="" httplog="" inspect-delay="" inter="" kubernetes-master="" kubernetes-master-pool="" local0="" log="" master01="" master02="" master03="" maxconn="" maxqueue="" mode="" monitor-interface="" monitor-uri="" option="" rise="" roundrobin="" server="" slowstart="" stats="" tcp="" tcp-check="" tcp-request="" tcplog="" timeout="" ulimit-n="" weight=""></eof>

Keepalived Configuration

# Master01 Keepalived configuration
cat > /etc/keepalived/keepalived.conf <<eof advert_int="" auth_pass="" auth_type="" authentication="" check-apiserver="" configuration="" dev="" enable_script_security="" eof="" eth0="" eth0:vip="" fall="" file="" for="" global_defs="" interface="" interval="" k8s_cluster_auth="" keepalived="" label="" lvs_main="" master="" mcast_src_ip="" pass="" priority="" rise="" root="" router_id="" script="" script_user="" state="" track_script="" vi_k8s="" virtual_ipaddress="" virtual_router_id="" vrrp_instance="" vrrp_script="" weight=""></eof>

# Master02 Keepalived configuration
cat > /etc/keepalived/keepalived.conf <<eof advert_int="" auth_pass="" auth_type="" authentication="" backup="" check-apiserver="" configuration="" dev="" enable_script_security="" eof="" eth0="" eth0:vip="" fall="" file="" for="" global_defs="" interface="" interval="" k8s_cluster_auth="" keepalived="" label="" lvs_backup="" mcast_src_ip="" pass="" priority="" rise="" root="" router_id="" script="" script_user="" state="" track_script="" vi_k8s="" virtual_ipaddress="" virtual_router_id="" vrrp_instance="" vrrp_script="" weight=""></eof>

# Master03 Keepalived configuration
cat > /etc/keepalived/keepalived.conf <<eof advert_int="" auth_pass="" auth_type="" authentication="" backup="" check-apiserver="" configuration="" dev="" enable_script_security="" eof="" eth0="" eth0:vip="" fall="" file="" for="" global_defs="" interface="" interval="" k8s_cluster_auth="" keepalived="" label="" lvs_backup="" mcast_src_ip="" pass="" priority="" rise="" root="" router_id="" script="" script_user="" state="" track_script="" vi_k8s="" virtual_ipaddress="" virtual_router_id="" vrrp_instance="" vrrp_script="" weight=""></eof>

Service Startup

# Enable and start services
systemctl enable keepalived && systemctl start keepalived && systemctl status keepalived
systemctl enable haproxy && systemctl start haproxy && systemctl status haproxy

Health Check Script

# Create health check script
cat > /etc/keepalived/check_apiserver.sh <<'EOF'
#!/bin/bash
error_count=0

for attempt in $(seq 1 3); do
    haproxy_status=$(pgrep haproxy)
    if [[ -z $haproxy_status ]]; then
        error_count=$((error_count + 1))
        sleep 1
        continue
    else
        error_count=0
        break
    fi
done

if [[ $error_count -ne 0 ]]; then
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi
EOF

chmod +x /etc/keepalived/check_apiserver.sh

Kubernetes Component Configuration

etcd Configuration

# Master01 etcd configuration
cat > /etc/etcd/etcd.conf.yml <<eof advertise-client-urls:="" auto-tls:="" cert-file:="" client-cert-auth:="" client-transport-security:="" cors:="" data-dir:="" debug:="" discovery-fallback:="" discovery-proxy:="" discovery-srv:="" discovery:="" election-timeout:="" enable-pprof:="" enable-v2:="" eof="" false="" force-new-cluster:="" heartbeat-interval:="" initial-advertise-peer-urls:="" initial-cluster-state:="" initial-cluster-token:="" initial-cluster:="" key-file:="" listen-client-urls:="" listen-peer-urls:="" log-outputs:="" log-package-levels:="" max-snapshots:="" max-wals:="" name:="" peer-client-cert-auth:="" peer-transport-security:="" proxy-dial-timeout:="" proxy-failure-wait:="" proxy-read-timeout:="" proxy-refresh-interval:="" proxy-write-timeout:="" proxy:="" quota-backend-bytes:="" snapshot-count:="" strict-reconfig-check:="" true="" trusted-ca-file:="" wal-dir:=""></eof>

# Master02 etcd configuration
cat > /etc/etcd/etcd.conf.yml <<eof advertise-client-urls:="" auto-tls:="" cert-file:="" client-cert-auth:="" client-transport-security:="" cors:="" data-dir:="" debug:="" discovery-fallback:="" discovery-proxy:="" discovery-srv:="" discovery:="" election-timeout:="" enable-pprof:="" enable-v2:="" eof="" false="" force-new-cluster:="" heartbeat-interval:="" initial-advertise-peer-urls:="" initial-cluster-state:="" initial-cluster-token:="" initial-cluster:="" key-file:="" listen-client-urls:="" listen-peer-urls:="" log-outputs:="" log-package-levels:="" max-snapshots:="" max-wals:="" name:="" peer-client-cert-auth:="" peer-transport-security:="" proxy-dial-timeout:="" proxy-failure-wait:="" proxy-read-timeout:="" proxy-refresh-interval:="" proxy-write-timeout:="" proxy:="" quota-backend-bytes:="" snapshot-count:="" strict-reconfig-check:="" true="" trusted-ca-file:="" wal-dir:=""></eof>

# Master03 etcd configuration
cat > /etc/etcd/etcd.conf.yml <<eof advertise-client-urls:="" auto-tls:="" cert-file:="" client-cert-auth:="" client-transport-security:="" cors:="" data-dir:="" debug:="" discovery-fallback:="" discovery-proxy:="" discovery-srv:="" discovery:="" election-timeout:="" enable-pprof:="" enable-v2:="" eof="" false="" force-new-cluster:="" heartbeat-interval:="" initial-advertise-peer-urls:="" initial-cluster-state:="" initial-cluster-token:="" initial-cluster:="" key-file:="" listen-client-urls:="" listen-peer-urls:="" log-outputs:="" log-package-levels:="" max-snapshots:="" max-wals:="" name:="" peer-client-cert-auth:="" peer-transport-security:="" proxy-dial-timeout:="" proxy-failure-wait:="" proxy-read-timeout:="" proxy-refresh-interval:="" proxy-write-timeout:="" proxy:="" quota-backend-bytes:="" snapshot-count:="" strict-reconfig-check:="" true="" trusted-ca-file:="" wal-dir:=""></eof>

# Create etcd systemd service on all master nodes
cat > /usr/lib/systemd/system/etcd.service <<eof after="network.target" and="" create="" daemon-reload="" description="etcd" documentation="https://coreos.com/etcd/docs/latest/" enable="" eof="" etcd="" execstart="/usr/local/bin/etcd" key-value="" limitnofile="65536" ln="" mkdir="" restart="on-failure" restartsec="10" start="" status="" store="" symlink="" systemctl="" type="notify" wantedby="multi-user.target"></eof>

# Verify etcd cluster health
export ETCDCTL_API=3
etcdctl --endpoints="10.0.0.171:2379,10.0.0.172:2379,10.0.0.173:2379" \
   --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem \
   --cert=/etc/kubernetes/pki/etcd/etcd.pem \
   --key=/etc/kubernetes/pki/etcd/etcd-key.pem \
   endpoint status --write-out=table

API Server Configuration

# Master01 API Server service
cat > /usr/lib/systemd/system/kube-apiserver.service <<eof after="network.target" api="" description="Kubernetes" documentation="https://github.com/kubernetes/kubernetes" eof="" execstart="/usr/local/bin/kube-apiserver" limitnofile="65535" restart="on-failure" restartsec="10s" server="" wantedby="multi-user.target"></eof>

# Master02 API Server service
cat > /usr/lib/systemd/system/kube-apiserver.service <<eof after="network.target" api="" description="Kubernetes" documentation="https://github.com/kubernetes/kubernetes" eof="" execstart="/usr/local/bin/kube-apiserver" limitnofile="65535" restart="on-failure" restartsec="10s" server="" wantedby="multi-user.target"></eof>

# Master03 API Server service
cat > /usr/lib/systemd/system/kube-apiserver.service <<eof after="network.target" api="" description="Kubernetes" documentation="https://github.com/kubernetes/kubernetes" eof="" execstart="/usr/local/bin/kube-apiserver" limitnofile="65535" restart="on-failure" restartsec="10s" server="" wantedby="multi-user.target"></eof>

# Start API Server on all master nodes
systemctl daemon-reload
systemctl enable --now kube-apiserver
systemctl status kube-apiserver

Controller Manager Configuration

# Controller Manager service on all master nodes
cat > /usr/lib/systemd/system/kube-controller-manager.service <<eof after="network.target" controller="" daemon-reload="" description="Kubernetes" documentation="https://github.com/kubernetes/kubernetes" enable="" eof="" execstart="/usr/local/bin/kube-controller-manager" kube-controller-manager="" manager="" restart="always" restartsec="10s" start="" status="" systemctl="" wantedby="multi-user.target"></eof>

Scheduler Configuration

# Scheduler service on all master nodes
cat > /usr/lib/systemd/system/kube-scheduler.service <<eof after="network.target" daemon-reload="" description="Kubernetes" documentation="https://github.com/kubernetes/kubernetes" enable="" eof="" execstart="/usr/local/bin/kube-scheduler" kube-scheduler="" restart="always" restartsec="10s" scheduler="" start="" status="" systemctl="" wantedby="multi-user.target"></eof>

TLS Bootstrapping Configuration

# Create bootstrap kubeconfig on Master01
cd /root/k8s-ha-install/bootstrap

kubectl config set-cluster kubernetes \
   --certificate-authority=/etc/kubernetes/pki/ca.pem \
   --embed-certs=true \
   --server=https://10.0.0.200:16443 \
   --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config set-credentials tls-bootstrap-token-user \
   --token=c8ad9c.2e4d610cf3e7426e \
   --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config set-context tls-bootstrap-token-user@kubernetes \
   --cluster=kubernetes \
   --user=tls-bootstrap-token-user \
   --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config use-context tls-bootstrap-token-user@kubernetes \
   --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

# Copy admin kubeconfig for cluster management
mkdir -p /root/.kube
cp /etc/kubernetes/admin.kubeconfig /root/.kube/config

# Verify cluster components
kubectl get cs

# Create bootstrap secrets and RBAC resources
cd /root/k8s-ha-install/bootstrap
kubectl create -f bootstrap.secret.yaml

Worker Node Configuration

# Distribute certificates from Master01 to all nodes
cd /etc/kubernetes/
for node in master02 master03 node01; do
    ssh $node mkdir -p /etc/kubernetes/pki
    for cert in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do
        scp /etc/kubernetes/$cert $node:/etc/kubernetes/${cert}
    done
done

# Create required directories on all nodes
mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/

# Create kubelet service on all nodes
cat > /usr/lib/systemd/system/kubelet.service <<eof description="Kubernetes" documentation="https://github.com/kubernetes/kubernetes" eof="" execstart="/usr/local/bin/kubelet" kubelet="" restart="always" restartsec="10" startlimitinterval="0" wantedby="multi-user.target"></eof>

# Configure kubelet service for Containerd runtime
cat > /etc/systemd/system/kubelet.service.d/10-kubelet.conf <<eof environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig" eof="" execstart="ExecStart=/usr/local/bin/kubelet"></eof>

# Configure kubelet service for Docker runtime
cat > /etc/systemd/system/kubelet.service.d/10-kubelet.conf <<eof environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig" eof="" execstart="ExecStart=/usr/local/bin/kubelet"></eof>

# Create kubelet configuration on all nodes
cat > /etc/kubernetes/kubelet-conf.yml <<eof address:="" all="" anonymous:="" apiversion:="" application="" authentication:="" authorization:="" cacheauthorizedttl:="" cachettl:="" cacheunauthorizedttl:="" cgroupdriver:="" cgroupsperqos:="" clientcafile:="" cluster.local="" clusterdns:="" clusterdomain:="" containerlogmaxfiles:="" containerlogmaxsize:="" contenttype:="" cpucfsquota:="" cpumanagerpolicy:="" cpumanagerreconcileperiod:="" daemon-reload="" enable="" enablecontrollerattachdetach:="" enabled:="" enabledebugginghandlers:="" enforcenodeallocatable:="" eof="" eventburst:="" eventrecordqps:="" evictionhard:="" evictionpressuretransitionperiod:="" failswapon:="" false="" filecheckfrequency:="" hairpinmode:="" healthzbindaddress:="" healthzport:="" httpcheckfrequency:="" imagefs.available:="" imagegchighthresholdpercent:="" imagegclowthresholdpercent:="" imageminimumgcage:="" iptablesdropbit:="" iptablesmasqueradebit:="" kind:="" kubeapiburst:="" kubeapiqps:="" kubelet="" kubelet.config.k8s.io="" kubeletconfiguration="" makeiptablesutilchains:="" maxopenfiles:="" maxpods:="" memory.available:="" mode:="" nodefs.available:="" nodefs.inodesfree:="" nodes="" nodestatusupdatefrequency:="" none="" on="" oomscoreadj:="" podpidslimit:="" pods="" port:="" promiscuous-bridge="" readonlyport:="" registryburst:="" registrypullqps:="" resolvconf:="" rotatecertificates:="" runtimerequesttimeout:="" serializeimagepulls:="" start="" staticpodpath:="" status="" streamingconnectionidletimeout:="" syncfrequency:="" systemctl="" systemd="" true="" volumestatsaggperiod:="" webhook="" webhook:="" x509:=""></eof>

# Verify node registration
kubectl get nodes

Kube-Proxy Configuration

# Generate kube-proxy certificate and kubeconfig
cd /root/k8s-ha-install/pki
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy

kubectl config set-cluster kubernetes \
   --certificate-authority=/etc/kubernetes/pki/ca.pem \
   --embed-certs=true \
   --server=https://10.0.0.200:16443 \
   --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

kubectl config set-credentials system:kube-proxy \
   --client-certificate=/etc/kubernetes/pki/kube-proxy.pem \
   --client-key=/etc/kubernetes/pki/kube-proxy-key.pem \
   --embed-certs=true \
   --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

kubectl config set-context system:kube-proxy@kubernetes \
   --cluster=kubernetes \
   --user=system:kube-proxy \
   --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

kubectl config use-context system:kube-proxy@kubernetes \
   --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

# Distribute kube-proxy kubeconfig to all nodes
for node in master02 master03; do
    scp /etc/kubernetes/kube-proxy.kubeconfig $node:/etc/kubernetes/kube-proxy.kubeconfig
done

for node in node01; do
    scp /etc/kubernetes/kube-proxy.kubeconfig $node:/etc/kubernetes/kube-proxy.kubeconfig
done

# Create kube-proxy service on all nodes
cat > /usr/lib/systemd/system/kube-proxy.service <<eof after="network.target" description="Kubernetes" documentation="https://github.com/kubernetes/kubernetes" eof="" execstart="/usr/local/bin/kube-proxy" kube="" proxy="" restart="always" restartsec="10s" wantedby="multi-user.target"></eof>

# Create kube-proxy configuration
cat > /etc/kubernetes/kube-proxy.yml <<eof acceptcontenttypes:="" all="" apiversion:="" application="" bindaddress:="" burst:="" clientconnection:="" clustercidr:="" configsyncperiod:="" conntrack:="" contenttype:="" daemon-reload="" enable="" enableprofiling:="" eof="" false="" healthzbindaddress:="" hostnameoverride:="" iptables:="" ipvs:="" kind:="" kube-proxy="" kubeconfig:="" kubeproxy.config.k8s.io="" kubeproxyconfiguration="" masqueradeall:="" masqueradebit:="" max:="" maxpercore:="" metricsbindaddress:="" min:="" minsyncperiod:="" mode:="" nodeportaddresses:="" nodes="" null="" on="" oomscoreadj:="" portrange:="" qps:="" scheduler:="" start="" status="" syncperiod:="" systemctl="" tcpclosewaittimeout:="" tcpestablishedtimeout:="" true="" udpidletimeout:=""></eof>

Calico Network Plugin Installation

# Configure Pod CIDR in Calico manifest
cd /root/k8s-ha-install/calico/
sed "s#POD_CIDR#172.16.0.0/16#g" calico.yaml | grep "172"

# Apply manifest changes
sed -i "s#POD_CIDR#172.16.0.0/16#g" calico.yaml

# Deploy Calico
kubectl apply -f calico.yaml

# Monitor deployment
kubectl get pods -n kube-system

CoreDNS Installation

# Determine DNS service IP
cd /root/k8s-ha-install/
DNS_SERVICE_IP=$(kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}')0
echo "DNS Service IP: $DNS_SERVICE_IP"

# Update CoreDNS manifest
sed "s#KUBEDNS_SERVICE_IP#${DNS_SERVICE_IP}#g" CoreDNS/coredns.yaml | grep "10.96"
sed -i "s#KUBEDNS_SERVICE_IP#${DNS_SERVICE_IP}#g" CoreDNS/coredns.yaml

# Deploy CoreDNS
kubectl create -f CoreDNS/coredns.yaml

# Verify CoreDNS pods
kubectl get pods -n kube-system | grep -i dns

# Alternative: Deploy latest CoreDNS from official repository
DNS_SERVICE_IP=$(kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}')0
git clone https://github.com/coredns/deployment.git
cd deployment/kubernetes
./deploy.sh -s -i ${DNS_SERVICE_IP} | kubectl apply -f -

# Verify deployment
kubectl get pods -n kube-system -l k8s-app=kube-dns

Metrics Server Installation

# Deploy Metrics Server
cd /root/k8s-ha-install/metrics-server
kubectl create -f .

# Verify Metrics Server
kubectl get pods -n kube-system | grep metrics-server

Dashboard Installation

# Deploy Kubernetes Dashboard
cd /root/k8s-ha-install/dashboard/
kubectl create -f .

Cluster Verification

# Verify all nodes are ready
kubectl get nodes

# Verify all system pods are running
kubectl get pods -A

# Check network configuration
kubectl get svc
kubectl get pods -A -o wide

# Test resource creation
kubectl create deployment connectivity-test --image=registry.cn-beijing.aliyuncs.com/dotbalo/debug-tools -- sleep 3600

# Verify pod execution
kubectl exec -it $(kubectl get pods -l app=connectivity-test -o jsonpath='{.items[0].metadata.name}') -- bash

# Test DNS resolution
nslookup kubernetes.default.svc.cluster.local
nslookup kube-dns.kube-system.svc.cluster.local

# Verify Service connectivity from all nodes
curl -k https://10.96.0.1:443
curl -udp @10.96.0.10:53

# Test Pod-to-Pod communication across namespaces
kubectl get pods -o wide
kubectl exec -it $(kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[0].metadata.name}') -- nslookup <target-pod-ip>

# Verify cross-node communication
for node in master02 master03 node01; do
    ssh $node ping -c 2 <pod-ip> && echo "From: $node"
done

Tags: kubernetes high-availability etcd containerd docker

Posted on Sun, 25 Oct 2026 16:28:24 +0000 by thebluebus

Freaks City