Kubernetes 1.18 Binary Installation with High Availability Setup

Deployment Overview

This guide covers the complete binary-based deployment of Kubernetes version 1.18 with high availability configuration. The setup includes a three-node etcd cluster, multi-master architecture, and load balancing infrastructure.

Software Requirements

Component Version/Source Purpose
CentOS 7.7+ Host operating system
Kubernetes Server v1.18.4 Core cluster components
etcd v3.4.9 Distributed key-value store
cfssl R1.2 Certificate generation toolset
Docker 19.03.9 Container runtime
CNI Plugins v0.8.6 Container networking

Deployment Topology

Hostname IP Address Role Services
k8s-master-1 172.17.17.85 Master/Node apiserver, controller, scheduler, etcd, kubelet, proxy
k8s-node5 172.17.17.86 Worker Node kubelet, proxy, etcd
k8s-node6 172.17.17.87 Worker Node kubelet, proxy, etcd
k8s-lb-1 172.17.17.81 Load Balancer Nginx, Keepalived
k8s-lb-2 172.17.17.84 Load Balancer Backup Nginx, Keepalived

System Preparation

Execute the following steps on all nodes before proceeding with the installation.

Initial System Configuration

# Update YUM repository and install dependencies
yum -y update
yum -y install wget epel-release

# Disable SELinux and configure firewall
sed -i 's/enforcing/disabled/' /etc/selinux/config
systemctl disable firewalld && systemctl stop firewalld

# Disable swap
sed -ri 's/.*swap.*/#&/' /etc/fstab
swapoff -a

# Configure hostname resolution
cat >> /etc/hosts << EOF
172.17.17.85 k8s-master-1
172.17.17.86 k8s-node5
172.17.17.87 k8s-node6
EOF

# Enable kernel modules for Kubernetes networking
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf

# Configure NTP time synchronization
yum install chrony -y
systemctl enable chronyd && systemctl start chronyd
timedatectl set-timezone Asia/Shanghai
timedatectl set-ntp yes

Certificate Infrastructure Setup

Using cfssl for PKI certificate management provides a cleaner approach than traditional OpenSSL configuration.

Installing cfssl Tools

mkdir -p ~/TLS/{etcd,k8s} && cd ~/TLS
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl*

etcd CA Configuraton

cd ~/TLS/etcd

# CA configuration file
cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
        "expiry": "87600h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      }
    }
  }
}
EOF

# CA certificate signing request
cat > ca-csr.json << EOF
{
  "CN": "etcd CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "ca": {
    "expiry": "87600h"
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing"
    }
  ]
}
EOF

# Generate CA certificate
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

etcd Server Certificate

cat > server-csr.json << EOF
{
  "CN": "etcd",
  "hosts": [
    "172.17.17.85",
    "172.17.17.86",
    "172.17.17.87"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

etcd Cluster Deployment

Creating etcd Directories

mkdir -p /data/etcd/{bin,cfg,ssl,data} -p
cd /data/etcd

# Download and extract etcd binaries
wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz
tar xf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/etcd* /data/etcd/bin/

Node-Specific Configuration

Configure the first node (172.17.17.85):

cat > /data/etcd/cfg/etcd.conf << EOF
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/data/etcd/data/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.17.17.85:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.17.17.85:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.17.17.85:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.17.17.85:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://172.17.17.85:2380,etcd-2=https://172.17.17.86:2380,etcd-3=https://172.17.17.87:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

Systemd Service Definition

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/data/etcd/cfg/etcd.conf
ExecStart=/data/etcd/bin/etcd \
  --cert-file=/data/etcd/ssl/server.pem \
  --key-file=/data/etcd/ssl/server-key.pem \
  --peer-cert-file=/data/etcd/ssl/server.pem \
  --peer-key-file=/data/etcd/ssl/server-key.pem \
  --trusted-ca-file=/data/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/data/etcd/ssl/ca.pem \
  --logger=zap
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

# Copy certificates
mv ~/TLS/etcd/*pem /data/etcd/ssl/

# Start etcd service
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

Deploying Remaining etcd Nodes

Copy the etcd installation to other nodes:

# From master node to node5
scp -r /data/etcd/ root@172.17.17.86:/data/
scp /usr/lib/systemd/system/etcd.service root@172.17.17.86:/usr/lib/systemd/system/

# From master node to node6
scp -r /data/etcd/ root@172.17.17.87:/data/
scp /usr/lib/systemd/system/etcd.service root@172.17.17.87:/usr/lib/systemd/system/

Modify the configuration file on each node to reflect its identity:

# For node5 (/data/etcd/cfg/etcd.conf)
ETCD_NAME="etcd-2"
ETCD_LISTEN_PEER_URLS="https://172.17.17.86:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.17.17.86:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.17.17.86:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.17.17.86:2379"

# For node6 (/data/etcd/cfg/etcd.conf)
ETCD_NAME="etcd-3"
ETCD_LISTEN_PEER_URLS="https://172.17.17.87:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.17.17.87:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.17.17.87:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.17.17.87:2379"

# Start services on remaining nodes
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

Verifying etcd Cluster Health

ETCDCTL_API=3 /data/etcd/bin/etcdctl \
  --cacert=/data/etcd/ssl/ca.pem \
  --cert=/data/etcd/ssl/server.pem \
  --key=/data/etcd/ssl/server-key.pem \
  --endpoints="https://172.17.17.85:2379,https://172.17.17.86:2379,https://172.17.17.87:2379" \
  endpoint health

Expected output indicates all nodes are healthy:

https://172.17.17.85:2379 is healthy: successfully committed proposal
https://172.17.17.86:2379 is healthy: successfully committed proposal
https://172.17.17.87:2379 is healthy: successfully committed proposal

Docker Installation

Install Docker Engine on all nodes before proceeding with Kubernetes components.

# Download and extract Docker binaries
wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz
tar xf docker-19.03.9.tgz
mv docker/* /usr/bin/

# Create systemd service file
cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target
EOF

# Configure Docker with registry mirrors
mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
  "graph": "/data/docker",
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF

# Start Docker service
systemctl daemon-reload
systemctl restart docker
systemctl enable docker

Kubernetes Master Components

Kubernetes CA Configuration

cd ~/TLS/k8s

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "expiry": "87600h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json << EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

API Server Certificate

cat > server-csr.json << EOF
{
  "CN": "kubernetes",
  "hosts": [
    "10.0.0.1",
    "127.0.0.1",
    "172.17.17.85",
    "172.17.17.86",
    "172.17.17.87",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

Installing Kubernetes Binaries

mkdir -p /data/kubernetes/{cfg,bin,ssl,logs}

# Download Kubernetes server binaries
wget https://dl.k8s.io/v1.18.4/kubernetes-server-linux-amd64.tar.gz
tar xf kubernetes-server-linux-amd64.tar.gz

# Copy master components
cp kubernetes/server/bin/kube-apiserver /data/kubernetes/bin/
cp kubernetes/server/bin/kube-controller-manager /data/kubernetes/bin/
cp kubernetes/server/bin/kube-scheduler /data/kubernetes/bin/
cp kubernetes/server/bin/kubectl /usr/bin/

API Server Configuration

cat > /data/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/data/kubernetes/logs \\
--etcd-servers=https://172.17.17.85:2379,https://172.17.17.86:2379,https://172.17.17.87:2379 \\
--bind-address=172.17.17.85 \\
--secure-port=6443 \\
--advertise-address=172.17.17.85 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/data/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/data/kubernetes/ssl/server.pem \\
--kubelet-client-key=/data/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/data/kubernetes/ssl/server.pem \\
--tls-private-key-file=/data/kubernetes/ssl/server-key.pem \\
--client-ca-file=/data/kubernetes/ssl/ca.pem \\
--service-account-key-file=/data/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/data/etcd/ssl/ca.pem \\
--etcd-certfile=/data/etcd/ssl/server.pem \\
--etcd-keyfile=/data/etcd/ssl/server-key.pem \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/data/kubernetes/logs/k8s-audit.log"
EOF

Bootstrap Token Configuration

cat > /data/kubernetes/cfg/token.csv << EOF
2b4b65d2e33e24dc0beafddda6dd4b23,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

# Generate token if needed
# head -c 16 /dev/urandom | od -An -t x | tr -d ' '

API Server Systemd Service

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/data/kubernetes/cfg/kube-apiserver.conf
ExecStart=/data/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# Copy certificates
mv ~/TLS/k8s/*pem /data/kubernetes/ssl/

# Start API server
systemctl daemon-reload
systemctl start kube-apiserver
systemctl enable kube-apiserver

# Grant kubelet-bootstrap permissions
kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \
  --user=kubelet-bootstrap

Controller Manager Configuration

cat > /data/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/data/kubernetes/logs \\
--leader-elect=true \\
--master=127.0.0.1:8080 \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/data/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/data/kubernetes/ssl/ca-key.pem \\
--root-ca-file=/data/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/data/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s"
EOF

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/data/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/data/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start kube-controller-manager
systemctl enable kube-controller-manager

Scheduler Configuration

cat > /data/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/data/kubernetes/logs \\
--leader-elect \\
--master=127.0.0.1:8080 \\
--bind-address=127.0.0.1"
EOF

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/data/kubernetes/cfg/kube-scheduler.conf
ExecStart=/data/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start kube-scheduler
systemctl enable kube-scheduler

# Verify cluster components
kubectl get cs

Worker Node Deployment

Kubelet and Proxy Installation

mkdir -p /data/kubernetes/{cfg,bin,ssl,logs}

# Copy worker binaries from downloaded package
cp kubernetes/server/bin/kube-proxy /data/kubernetes/bin/
cp kubernetes/server/bin/kubelet /data/kubernetes/bin/

# Copy CA certificate
cp /data/kubernetes/ssl/ca.pem /data/kubernetes/ssl/

Kubelet Configuration

cat > /data/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/data/kubernetes/logs \\
--hostname-override=k8s-master-1 \\
--network-plugin=cni \\
--kubeconfig=/data/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/data/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/data/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/data/kubernetes/ssl \\
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"
EOF

Kubelet Parameters File

cat > /data/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
  - 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /data/kubernetes/ssl/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100M
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

Bootstrap Kubeconfig Generation

KUBE_APISERVER="https://172.17.17.85:6443"
TOKEN="2b4b65d2e33e24dc0beafddda6dd4b23"

kubectl config set-cluster kubernetes \
  --certificate-authority=/data/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

kubectl config set-credentials kubelet-bootstrap \
  --token=${TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

kubectl config use-context default \
  --kubeconfig=bootstrap.kubeconfig

cp bootstrap.kubeconfig /data/kubernetes/cfg/

Kubelet Systemd Service

cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service

[Service]
EnvironmentFile=/data/kubernetes/cfg/kubelet.conf
ExecStart=/data/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet

Approving Node Certificate Request

# Check pending CSR
kubectl get csr

# Approve the certificate
kubectl certificate approve <CSR-NAME>

# Verify node status
kubectl get nodes

Kube-Proxy Configuration

cat > /data/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/data/kubernetes/logs \\
--config=/data/kubernetes/cfg/kube-proxy-config.yml"
EOF

cat > /data/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /data/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-master-1
clusterCIDR: 10.0.0.0/24
EOF

Kube-Proxy Certificate and Kubeconfig

cd ~/TLS/k8s

cat > kube-proxy-csr.json << EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

# Generate kubeconfig
KUBE_APISERVER="https://172.17.17.85:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/data/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default \
  --kubeconfig=kube-proxy.kubeconfig

cp kube-proxy.kubeconfig /data/kubernetes/cfg/

Kube-Proxy Systemd Service

cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=/data/kubernetes/cfg/kube-proxy.conf
ExecStart=/data/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start kube-proxy
systemctl enable kube-proxy

CNI Network Deployment

Enstalling CNI Plugins

wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
mkdir /opt/cni/bin -p
tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin/

Deploying Flannel Network

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml
kubectl apply -f kube-flannel.yml

# Verify node status
kubectl get nodes

API Server to Kubelet Authorization

cat > apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

kubectl apply -f apiserver-to-kubelet-rbac.yaml

Adding Worker Nodes

Copy required files to new worker nodes:

# Copy binaries
scp /data/kubernetes/bin/kubelet 172.17.17.86:/data/kubernetes/bin/
scp /data/kubernetes/bin/kubelet 172.17.17.87:/data/kubernetes/bin/
scp /data/kubernetes/bin/kube-proxy 172.17.17.86:/data/kubernetes/bin/
scp /data/kubernetes/bin/kube-proxy 172.17.17.87:/data/kubernetes/bin/

# Copy CNI plugins
scp -rp /opt/cni/ 172.17.17.86:/opt/
scp -rp /opt/cni/ 172.17.17.87:/opt/

# Copy certificates
scp /data/kubernetes/ssl/ca.pem 172.17.17.86:/data/kubernetes/ssl/
scp /data/kubernetes/ssl/ca.pem 172.17.17.87:/data/kubernetes/ssl/

# Copy configuration files
scp /data/kubernetes/cfg/kube-proxy* 172.17.17.86:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/kube-proxy* 172.17.17.87:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/kubelet* 172.17.17.86:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/kubelet* 172.17.17.87:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/bootstrap.kubeconfig 172.17.17.86:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/bootstrap.kubeconfig 172.17.17.87:/data/kubernetes/cfg/

# Copy service files
scp /usr/lib/systemd/system/kubelet.service 172.17.17.86:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/kubelet.service 172.17.17.87:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/kube-proxy.service 172.17.17.86:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/kube-proxy.service 172.17.17.87:/usr/lib/systemd/system/

Remove auto-generated files and configure host-specific settings:

# On each new node, remove generated files
rm -rf /data/kubernetes/cfg/kubelet.kubeconfig
rm -f /data/kubernetes/ssl/kubelet*

# Modify kubelet.conf - change hostname-override
--hostname-override=k8s-node5

# Modify kube-proxy-config.yml - change hostnameOverride
hostnameOverride: k8s-node5

# Start services
systemctl daemon-reload
systemctl start kubelet
systemctl start kube-proxy
systemctl enable kubelet
systemctl enable kube-proxy

Approve new node certificates on master:

kubectl get csr
kubectl certificate approve <node-csr-name>
kubectl get nodes

Dashboard and CoreDNS

Kubernetes Dashboard

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

# Modify service type to NodePort
vim recommended.yaml
# Find the Service section and add:
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001

kubectl apply -f recommended.yaml
kubectl get pods,svc -n kubernetes-dashboard

Create admin service account for dashboard access:

kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system \
  $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

CoreDNS Deployment

kubectl apply -f coredns.yaml
kubectl get pods -n kube-system

# If network issues occur
ifconfig cni0 down
ip link delete cni0

Test DNS resolution:

kubectl run -it --rm dns-test --image=busybox:1.28.4 sh
/ # nslookup kubernetes

High Availability Architecture

Configure load balancing for multi-master Kubernetes deployment.

Nginx Load Balancer Configuration

yum install epel-release -y
yum install nginx keepalived -y

cat > /etc/nginx/nginx.conf << EOF
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

events {
  worker_connections: 1024;
}

stream {
  log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
  access_log /var/log/nginx/k8s-access.log main;
  
  upstream k8s-apiserver {
    server 172.17.17.85:6443;
    server 172.17.17.84:6443;
  }
  
  server {
    listen 6443;
    proxy_pass k8s-apiserver;
  }
}

http {
  log_format main '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';
  access_log /var/log/nginx/access.log main;
  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;
  types_hash_max_size 2048;
  include /etc/nginx/mime.types;
  default_type application/octet-stream;
  server {
    listen 80;
    server_name _;
  }
}
EOF

Keepalived Configuration (Master)

cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
  notification_email {
    admin@example.com
    failover@example.com
    sysadmin@example.com
  }
  notification_email_from admin@example.com
  smtp_server 127.0.0.1
  smtp_connect_timeout 30
  router_id NGINX_MASTER
}

vrrp_script check_nginx {
  script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
  state MASTER
  interface enp9s0
  virtual_router_id 51
  priority 100
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  virtual_ipaddress {
    172.17.17.79/24
  }
  track_script {
    check_nginx
  }
}
EOF

# Health check script
cat > /etc/keepalived/check_nginx.sh << EOF
#!/bin/bash
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ]; then
  exit 1
else
  exit 0
fi
EOF
chmod +x /etc/keepalived/check_nginx.sh

Keepalived Configuration (Backup)

cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
  notification_email {
    admin@example.com
    failover@example.com
    sysadmin@example.com
  }
  notification_email_from admin@example.com
  smtp_server 127.0.0.1
  smtp_connect_timeout 30
  router_id NGINX_BACKUP
}

vrrp_script check_nginx {
  script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
  state BACKUP
  interface enp9s0
  virtual_router_id 51
  priority 90
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  virtual_ipaddress {
    172.17.17.79/24
  }
  track_script {
    check_nginx
  }
}
EOF

Start Load Balancer Services

systemctl daemon-reload
systemctl start nginx
systemctl start keepalived
systemctl enable nginx
systemctl enable keepalived

# Verify VIP is assigned
ip addr show enp9s0

Update Worker Nodes for Load Balancer

Modify worker node configurations to use the virtual IP:

cd /data/kubernetes/cfg/
sed -i 's/172.17.17.85/172.17.17.79/g' *
systemctl restart kubelet
systemctl restart kube-proxy

Verification Commands

Final cluster status verification:

kubectl get cs
kubectl get nodes
kubectl get pods --all-namespaces

Expected output shows all components healthy with all nodes in Ready state.

Tags: kubernetes k8s etcd high-availability nginx

Posted on Wed, 01 Jul 2026 16:01:38 +0000 by melittle

Hot Tags

©2026 Freaks City