Prerequisites

Begin by preparing the infrastructure. The simplest approach is to provition virtual machines through a public cloud provider. Alternatively, physical servers can be used if available. These machines must meet the following criteria:

• Compatible with Docker installation requirements, such as 64-bit Linux OS and kernel version 3.10 or higher;
• Support for x86+ or ARM+ architectures;
• Network connectivity between machines to enable container communication;
• Internet access (via VPN where necesary) for pulling container images;
• Access to gcr.io and quay.io registries for specific image pulls;
• Minimum 2 CPU cores and 8 GB RAM per machine, though smaller specs will work with limited Pod scheduling capacity;
• At least 30 GB of available disk space for Docker images and logs.

Installing kubeadm and Docker

The kubeadm tool simplifies Kubernetes deployment. First, add the official repository:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

Add the Kubernetes repository to /etc/apt/sources.list.d/kubernetes.list:

deb http://apt.kubernetes.io/ kubernetes-xenial main

Update package indexes:

apt-get update

Install Docker CE version 17.03.3:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get update
apt-get install docker-ce=17.03.3~ce-0~ubuntu-xenial

Install Kubernetes components:

apt install kubelet=1.11.1-00
apt install kubectl=1.11.1-00
apt install kubeadm=1.11.1-00

Setting Up the Master Node

Deploy the Master node using kubeadm. For enhanced functionality, use a configuration file:

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
controllerManagerExtraArgs:
  horizontal-pod-autoscaler-use-rest-clients: "true"
  horizontal-pod-autoscaler-sync-period: "10s"
  node-monitor-grace-period: "10s"
apiServerExtraArgs:
  runtime-config: "api/all=true"
kubernetesVersion: "v1.11.1"

Apply this configuration with:

kubeadm init --config kubeadm.yaml

After completion, kubeadm provides a join command for worker nodes:

kubeadm join 209.250.244.253:6443 --token 4srvlc.6ug9t7ikb3ao0m06 --discovery-token-ca-cert-hash sha256:b71d09fee04a19c53787b44bd18df39b50256de6beeee90f3acc2665948d18b8

Save this command for later use. Additionally, configure local access:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Check node status:

kubectl get nodes

The output shows the Master node as NotReady. This occurs because no network plugin has been installed yet. Enspect the node details:

kubectl describe node master

Verify system pods in the kube-system namespace:

kubectl get pods -n kube-system

Pods like CoreDNS remain in Pending state due to missing networking.

Installing a Network Plugin

Deploy a network plugin using kubectl apply:

kubectl apply -f weave-daemonset-k8s-1.6.yaml

Check pod statuses:

kubectl get pods -n kube-system

All system pods should now be running, including the Weave plugin daemonset. Kubernetes supports various plugins via the Container Network Interface (CNI), such as Flannel, Calico, Canal, and Romana.

Adding Worker Nodes

Worker nodes require the same setup as the Master. Install required packages on each node:

1. Install Docker and kubeadm
2. Run the join command provided earlier:

kubeadm join 209.250.244.253:6443 --token 4srvlc.6ug9t7ikb3ao0m06 --discovery-token-ca-cert-hash sha256:b71d09fee04a19c53787b44bd18df39b50256de6beeee90f3acc2665948d18b8

Configuring Master Node Scheduling

By default, the Master node does not schedule user Pods. This behavior uses Taints and Tolerations:

Add a Taint to prevent scheduling:

kubectl taint nodes node1 foo=bar:NoSchedule

To allow Pods to run on the Master, define a Toleration in the Pod manifest:

spec:
  tolerations:
  - key: "foo"
    operator: "Equal"
    value: "bar"
    effect: "NoSchedule"

For the default Master Taint, use:

spec:
  tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"

To remove the Taint completely:

kubectl taint nodes --all node-role.kubernetes.io/master-

Deploying the Dashboard

Download and modify the dashboard manifest to expose it via NodePort:

wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.0/src/deploy/recommended/kubernetes-dashboard.yaml
kubectl apply -f kubernetes-dashboard.yaml

Verify the deployment:

kubectl get pods -n kube-system

Access the dashboard locally via proxy or Ingress for external access.

Installing Storage Plugin

Deploy Rook for persistent storage:

kubectl apply -f https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/ceph/operator.yaml
kubectl apply -f https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/ceph/cluster.yaml

Monitor the deployment:

kubectl get pods --all-namespaces
kubectl get pods -n rook-ceph-system
kubectl get pods -n rook-ceph

Rook manages storage through Persistent Volumes and Claims, providing enterprise-grade features like backup, scaling, and monitoring.