Infrastructure Architecture

Operating System	Kernel	IP Address	Hostname	Role
Ubuntu 22.04 LTS	5.15.0	10.0.50.11	control-plane-1	Control Plane
Ubuntu 22.04 LTS	5.15.0	10.0.50.12	control-plane-2	Control Plane
Ubuntu 22.04 LTS	5.15.0	10.0.50.13	control-plane-3	Control Plane
Ubuntu 22.04 LTS	5.15.0	10.0.50.21	worker-1	Worker
Ubuntu 22.04 LTS	5.15.0	10.0.50.22	worker-2	Worker
Ubuntu 22.04 LTS	5.15.0	10.0.50.23	worker-3	Worker

Virtual IP: 10.0.50.10

Node Preparation

Configure host identification and remote access:

hostnamectl set-hostname control-plane-1

apt update
apt install -y openssh-server vim
sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
systemctl restart sshd

Install system dependencies:

apt install -y apt-transport-https ca-certificates curl gnupg lsb-release \
  net-tools tcpdump ntp bridge-utils tree zip wget iftop ethtool dnsutils

Configure Kubernetes package repository:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.24/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.24/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt update

Disable swap and configure kernel parameters:

swapoff -a
sed -i '/swap/d' /etc/fstab

modprobe overlay
modprobe br_netfilter

cat <<EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sysctl --system

Install Kubernetes components:

apt install -y kubelet=1.24.3-00 kubeadm=1.24.3-00 kubectl=1.24.3-00
apt-mark hold kubelet kubeadm kubectl

Container Runtime Installation

Deploy containerd with CNI support:

cd /tmp
wget https://github.com/containerd/containerd/releases/download/v1.6.6/cri-containerd-cni-1.6.6-linux-amd64.tar.gz
tar Cxzvf / cri-containerd-cni-1.6.6-linux-amd64.tar.gz
mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml > /dev/null

Modify containerd configuration for systemd cgroup driver and image registry:

sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sed -i 's|sandbox_image = ".*"|sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.7"|' /etc/containerd/config.toml

Configure private registry authentication (adjust for your environment):

# Append to /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.0.50.100"]
  endpoint = ["http://10.0.50.100"]

[plugins."io.containerd.grpc.v1.cri".registry.configs."10.0.50.100".tls]
  insecure_skip_verify = true

[plugins."io.containerd.grpc.v1.cri".registry.configs."10.0.50.100".auth]
  username = "admin"
  password = "RegistryPassword123"

Start containerd service:

systemctl daemon-reload
systemctl enable --now containerd
systemctl status containerd --no-pager

Install nerdctl for debugging:

wget https://github.com/containerd/nerdctl/releases/download/v0.22.2/nerdctl-0.22.2-linux-amd64.tar.gz
tar Cxzvf /usr/local/bin nerdctl-0.22.2-linux-amd64.tar.gz

Control Plane Load Balancing with kube-vip

Establish environment variables for VIP configuration:

export CONTROL_PLANE_VIP=10.0.50.10
export INTERFACE_NAME=eth0

Generate the kube-vip static pod manifest on each control plane node:

ctr image pull docker.io/plndr/kube-vip:v0.5.0

ctr run --rm --net-host docker.io/plndr/kube-vip:v0.5.0 vip \
  /kube-vip manifest pod \
  --interface ${INTERFACE_NAME} \
  --vip ${CONTROL_PLANE_VIP} \
  --controlplane \
  --services \
  --arp \
  --leaderElection | tee /etc/kubernetes/manifests/kube-vip.yaml

Repeat the manifest generation on all control plane nodes (control-plane-2 and control-plane-3).

Cluster Initialization

Create the kubeadm configuration file:

# cluster-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 9a08jv.9a08jv9a08jv9a08
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
localAPIEndpoint:
  advertiseAddress: 10.0.50.11
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: control-plane-1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: 1.24.3
controlPlaneEndpoint: 10.0.50.10:6443
apiServer:
  certSANs:
  - 127.0.0.1
  - 10.0.50.10
  - control-plane-1
  - control-plane-2
  - control-plane-3
  - 10.0.50.11
  - 10.0.50.12
  - 10.0.50.13
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
scheduler: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.32.0.0/12
  podSubnet: 10.200.0.0/16
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
clusterDNS:
- 10.32.0.10
clusterDomain: cluster.local
rotateCertificates: true
staticPodPath: /etc/kubernetes/manifests

Initialize the cluster without kube-proxy (Cilium will handle this):

kubeadm init --skip-phases=addon/kube-proxy --upload-certs --config cluster-config.yaml

Join additional control plane nodes:

kubeadm join 10.0.50.10:6443 --token 9a08jv.9a08jv9a08jv9a08 \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <key>

Configure kubectl:

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

Cilium CNI Installation

Install Helm:

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | tee /usr/share/keyrings/helm.gpg > /dev/null
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | tee /etc/apt/sources.list.d/helm-stable-debian.list
apt update
apt install helm

Add Cilium repository:

helm repo add cilium https://helm.cilium.io/
helm repo update

Deploy Cilium in kube-proxy replacement mode:

helm install cilium cilium/cilium --version 1.12.1 \
  --namespace kube-system \
  --set kubeProxyReplacement=strict \
  --set k8sServiceHost=10.0.50.10 \
  --set k8sServicePort=6443 \
  --set bpf.masquerade=true \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set prometheus.enabled=true \
  --set operator.prometheus.enabled=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}"

Verify Cilium status:

cilium status

Multus Multi-Network Configuration

Deploy Multus CNI:

# multus-ds.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: network-attachment-definitions.k8s.cni.cncf.io
spec:
  group: k8s.cni.cncf.io
  scope: Namespaced
  names:
    plural: network-attachment-definitions
    singular: network-attachment-definition
    kind: NetworkAttachmentDefinition
    shortNames:
    - net-attach-def
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          description: 'NetworkAttachmentDefinition is a CRD schema for attaching pods to networks'
          type: object
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              type: object
              properties:
                config:
                  type: string
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: multus-cni
rules:
- apiGroups: ["k8s.cni.cncf.io"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: [""]
  resources: ["pods", "pods/status"]
  verbs: ["get", "update"]
- apiGroups: ["", "events.k8s.io"]
  resources: ["events"]
  verbs: ["create", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: multus-cni
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: multus-cni
subjects:
- kind: ServiceAccount
  name: multus
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: multus
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: multus-ds
  namespace: kube-system
  labels:
    app: multus
spec:
  selector:
    matchLabels:
      app: multus
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: multus
    spec:
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      - operator: Exists
        effect: NoExecute
      serviceAccountName: multus
      initContainers:
      - name: install-multus-binary
        image: ghcr.io/k8snetworkplumbingwg/multus-cni:stable
        command: ["cp", "/usr/src/multus-cni/bin/multus", "/host/opt/cni/bin/multus"]
        resources:
          requests:
            cpu: "10m"
            memory: "15Mi"
        securityContext:
          privileged: true
        volumeMounts:
        - name: cnibin
          mountPath: /host/opt/cni/bin
          mountPropagation: Bidirectional
      containers:
      - name: kube-multus
        image: ghcr.io/k8snetworkplumbingwg/multus-cni:stable
        command: ["/entrypoint.sh"]
        args:
        - "--multus-conf-file=auto"
        - "--cni-version=0.3.1"
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        volumeMounts:
        - name: cni
          mountPath: /host/etc/cni/net.d
        - name: cnibin
          mountPath: /host/opt/cni/bin
      terminationGracePeriodSeconds: 10
      volumes:
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: cnibin
        hostPath:
          path: /opt/cni/bin

Apply the Multus daemonset:

kubectl apply -f multus-ds.yaml

Create a macvlan network attachment definition:

# macvlan-net.yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-net
  namespace: default
spec:
  config: |
    {
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "eth1",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.10.0/24",
        "rangeStart": "192.168.10.100",
        "rangeEnd": "192.168.10.200",
        "gateway": "192.168.10.1"
      }
    }

Deploy a test pod with multiple interfaces:

# multi-homed-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: multi-homed-client
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-net
spec:
  containers:
  - name: client
    image: alpine:latest
    command: ["/bin/sh", "-c", "sleep infinity"]
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"

Verify the pod has multiple network interfaces:

kubectl exec -it multi-homed-client -- ip addr