Prerequisites and Setup Guidelines

For Rook versions 1.3 and above, directory-based storage is no longer supported. You must attach raw, unformatted block devices to the host nodes for OSD provisioning. Ensure each participating node has a minimum of 2 CPU cores and 4 GB of RAM. Additionally, if your Kubernetes version is 1.19 or higher, the CSI snapshot controller must be deployed manually before proceeding.

Retrieving the Manifests

Fetch the required resource definitions from the official repository:

git clone --depth 1 --branch v1.14.7 https://github.com/rook/rook.git
cd rook/deploy/examples

Customizing the Operator

To bypass firewall restrictions pulling images from gcr.io, mirror the CSI components to an internal registry and update the operator.yaml configuration. Modify the rook-ceph-operator-config ConfigMap to point to your local registry. Ensure namespace isolation is enabled by adjusting the relevant flag.

kind: ConfigMap
apiVersion: v1
metadata:
  name: rook-ceph-operator-config
data:
  ROOK_LOG_LEVEL: "INFO"
  ROOK_CEPH_ALLOW_LOOP_DEVICES: "false"
  ROOK_CSI_ENABLE_CEPHFS: "true"
  ROOK_CSI_ENABLE_RBD: "true"
  ROOK_CSI_ENABLE_NFS: "false"
  ROOK_CSI_DISABLE_DRIVER: "false"
  CSI_ENABLE_ENCRYPTION: "false"
  CSI_DISABLE_HOLDER_PODS: "true"
  CSI_PROVISIONER_REPLICAS: "2"
  CSI_ENABLE_CEPHFS_SNAPSHOTTER: "true"
  CSI_ENABLE_RBD_SNAPSHOTTER: "true"
  ROOK_CSI_CEPH_IMAGE: "registry.mycorp.io/storage/cephcsi:v3.11.0"
  ROOK_CSI_REGISTRAR_IMAGE: "registry.mycorp.io/storage/csi-node-driver-registrar:v2.10.1"
  ROOK_CSI_RESIZER_IMAGE: "registry.mycorp.io/storage/csi-resizer:v1.10.1"
  ROOK_CSI_PROVISIONER_IMAGE: "registry.mycorp.io/storage/csi-provisioner:v4.0.1"
  ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.mycorp.io/storage/csi-snapshotter:v7.0.2"
  ROOK_CSI_ATTACHER_IMAGE: "registry.mycorp.io/storage/csi-attacher:v4.5.1"
  ROOK_ENABLE_DISCOVERY_DAEMON: "true"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rook-ceph-operator
  labels:
    app.kubernetes.io/name: rook-operator
    storage-layer: ceph
spec:
  selector:
    matchLabels:
      app: rook-ceph-operator
  strategy:
    type: Recreate
  replicas: 1
  template:
    metadata:
      labels:
        app: rook-ceph-operator
    spec:
      serviceAccountName: rook-ceph-system
      containers:
        - name: rook-ceph-operator
          image: registry.mycorp.io/storage/ceph:v1.14.7
          args: ["ceph", "operator"]
          securityContext:
            runAsNonRoot: true
            runAsUser: 1001
            runAsGroup: 1001
          env:
            - name: ROOK_CURRENT_NAMESPACE_ONLY
              value: "true"
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName

Initializing the Operator

Apply the foundational resources and the operator deployment:

kubectl create -f crds.yaml -f common.yaml -f operator.yaml

Wait until the operator pod and the discovery daemonsets transition to the Running state across all nodes before proceeding to cluster creation.

kubectl get pods -n rook-ceph
NAME                                  READY   STATUS    RESTARTS   AGE
rook-ceph-operator-7d8f9b6c4d-x2k9p   1/1     Running   0          2m
rook-discover-ab12c                   1/1     Running   0          1m
rook-discover-cd34e                   1/1     Running   0          1m
rook-discover-ef56g                   1/1     Running   0          1m

Provisioning the Storage Cluster

Node and Disk Configuration

Adjust the cluster.yaml manifest to explicitly define which nodes will host OSDs and specify the raw devices assigned to them. Verify the available unformatted disks using lsblk on the target hosts:

lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   80G  0 disk
├─sda1   8:1    0  500M  0 part /boot
└─sda2   8:2    0 79.5G  0 part /
vdb    252:16   0  200G  0 disk

Update the storage section of the cluster manifest to target these specific block devices:

storage:
  useAllNodes: false
  useAllDevices: false
  nodes:
    - name: "storage-node-01"
      devices:
        - name: "vdb"
    - name: "storage-node-02"
      devices:
        - name: "vdb"
    - name: "storage-node-03"
      devices:
        - name: "vdb"