Understanding cgroups

Before addressing the configuration issue, it's essential to understand what cgroups represents in the Linux kernel.

cgroupfs is a virtual filesystem type developed to provide user-facing operations for control groups. It presents the cgroup hierarchy to users, communicates kernel changes requested by users, and handles all cgroup queries and modifications through the filesystem interface.

Why systemd is Recommended

Kubernetes recommends using systemd as the cgroup driver instead of cgroupfs. The reasoning is straightforward: systemd serves as Kubernetes' built-in cgroup manager, handling resource allocation for each process across the cluster.

The problem arises when Docker defaults to cgroupfs while Kubernetes expects systemd. Runing two separate cgroup control managers simultaneously creates potential instability, particularly under resource pressure.

Identifying the Error

When the cgroup drivers are misaligned, the kubelet fails to start during cluster initialization. You can examine the detailed error output using:

journalctl -xeu kubelet

The error message appears as:

"Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""

To confirm the current Docker cgroup driver configuration:

docker info

Look for the Cgroup Driver line in the output. The default value will be cgroupfs.

Resolution

To align Docker with Kubernetes requirements, configure the Docker daemon on all master and worker nodes with the following settings.

Create or modify /etc/docker/daemon.json:

{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

After saving the configuration, restart the Docker service to apply the changes:

systemctl restart docker

Then restart the kubelet:

systemctl restart kubelet

Once these services are running with matching cgroup drivers, the kubelet should initialize successfully.