Best Practices for Kubernetes Management Platform with KubeSphere

Introduction

KubeSphere is an open-source distributed operating system built on top of Kubernetes, designed for cloud-native applications. It supports multi-cloud and multi-cluster management, provides full-stack IT automation capabilities, and simplifies DevOps workflows. The platform’s architecture enables seamless plug-and-play integration with third-party applications and cloud-native ecosystem components. KubeKey, an accompanying tool, assists in rapidly provisioning Kubernetes clusters on public clouds or data centers, supporting single-node, multi-node, cluster add-on installation, aswell as cluster upgrades and maintenance.

Environment Preparation

Three servers with 8 vCPUs, 16 GB RAM, 500 GB high-speed disk (master nodes).
Three servers with 16 vCPUs, 32 GB RAM, 500 GB high-speed disk (worker nodes).
Operating system: CentOS 7.
Disable SELinux and firewall.
Upgrade the system kernel (refer to appropriate documentation).

Install Dependencies

yum install socat conntrack ebtables ipset -y

Download KubeKey

export KKZONE=cn
curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.10 sh -
chmod +x kk
./kk create config --with-kubesphere v3.4.0 --with-kubernetes v1.23.10

Configure the Cluster (config-sample.yaml)

The configuration file defines the cluster topology, roles, network settings, and KubeSphere components. Below is a representative example with modified IP addresses and credentials for demonstration.

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: master1, address: 10.0.0.11, internalAddress: 10.0.0.11, user: root, password: "ChangeMe123"}
  - {name: master2, address: 10.0.0.12, internalAddress: 10.0.0.12, user: root, password: "ChangeMe123"}
  - {name: master3, address: 10.0.0.13, internalAddress: 10.0.0.13, user: root, password: "ChangeMe123"}
  - {name: node1, address: 10.0.0.14, internalAddress: 10.0.0.14, user: root, password: "ChangeMe123"}
  - {name: node2, address: 10.0.0.15, internalAddress: 10.0.0.15, user: root, password: "ChangeMe123"}
  - {name: node3, address: 10.0.0.16, internalAddress: 10.0.0.16, user: root, password: "ChangeMe123"}
  roleGroups:
    etcd:
    - master[1:3]
    control-plane: 
    - master[1:3]
    worker:
    - node[1:3]
  controlPlaneEndpoint:
    internalLoadbalancer: haproxy
    domain: lb.mycluster.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.23.10
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: docker
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    multusCNI:
      enabled: false
  registry:
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: ["https://docker.m.daocloud.io", "https://dockerproxy.com"]
    insecureRegistries: ["harbor.example.com"]
  addons: []

---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    version: v3.4.0
spec:
  persistence:
    storageClass: ""
  authentication:
    jwtSecret: ""
  etcd:
    monitoring: true
    endpointIps: localhost
    port: 2379
    tlsEnable: true
  common:
    core:
      console:
        enableMultiLogin: true
        port: 30880
        type: NodePort
    redis:
      enabled: false
      enableHA: false
      volumeSize: 2Gi
    openldap:
      enabled: false
      volumeSize: 2Gi
    minio:
      volumeSize: 20Gi
    monitoring:
      endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
      GPUMonitoring:
        enabled: false
    es:
      logMaxAge: 7
      elkPrefix: logstash
      basicAuth:
        enabled: false
    opensearch:
      enabled: true
      logMaxAge: 7
      opensearchPrefix: whizard
      basicAuth:
        enabled: true
        username: "admin"
        password: "admin"
      dashboard:
        enabled: false
  alerting:
    enabled: true
  auditing:
    enabled: true
  devops:
    enabled: false
  events:
    enabled: true
  logging:
    enabled: true
    logsidecar:
      enabled: true
      replicas: 2
  metrics_server:
    enabled: true
  multicluster:
    clusterRole: none
  network:
    networkpolicy:
      enabled: true
    ippool:
      type: calico
    topology:
      type: weave-scope
  servicemesh:
    enabled: true
    istio:
      components:
        ingressGateways:
        - name: istio-ingressgateway
          enabled: false
  edgeruntime:
    enabled: true
    kubeedge:
      enabled: true
      cloudCore:
        cloudHub:
          advertiseAddress:
            - "10.0.0.11"
        service:
          cloudhubNodePort: "30000"
          cloudhubQuicNodePort: "30001"
          cloudhubHttpsNodePort: "30002"
          cloudstreamNodePort: "30003"
          tunnelNodePort: "30004"
  terminal:
    timeout: 600

Deploy KubeSphere and Kubernetes

Run the cluster creation command and monitor the installation logs:

./kk create cluster -f config-sample.yaml
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

Once the deployment succeeds, you will see output similar to the following:

#####################################################
###              Welcome to KubeSphere!           ###
#####################################################
Console: http://10.0.0.11:30880
Account: admin
Password: P@88w0rd
NOTES:
  1. After you log into the console, please check the
     monitoring status of service components in
     the "Cluster Management". If any service is not
     ready, please wait patiently.
  2. Please change the default password after login.
#####################################################
https://kubesphere.io             2022-12-01 10:00:00
#####################################################

Enable Command Auto-completion for kubectl

yum -y install bash-completion
echo 'source <(kubectl completion bash)' >> ~/.bashrc

Alternatively, you can copy the completion script to the system-wide bash completion directory:

kubectl completion bash > /etc/bash_completion.d/kubectl

Freaks City