Use separate physical NICs for public and cluster networks to avoid contention. Ensure all OSD drives are identical in cappacity, speed, and firmware to prevent performance skew. For legacy drives, pre-configure them as RAID or ZFS volumes to standardize I/O characteristics.

System Preparation

Disable SELinux and firewall services to avoid interference with Ceph’s internal communication. While disabling fireawlls is acceptable in isolated environments, in production, open only required ports (e.g., 6789, 6800-7300) instead.

setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
systemctl stop firewalld && systemctl disable firewalld

SSH Key-Based Authentication

Generate SSH keys on the first node and distribute the public key to all cluster members for passwordless access.

ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

for node in ceph-node-2 ceph-node-3 ceph-node-4; do
  ssh-copy-id $node
done

Time Synchronization with Chrony

Ceph is sensitive to clock drift. Configure one node as the NTP server and others as clients.

# Install on all nodes
yum install -y chrony

# On ceph-node-1 (NTP server):
cat > /etc/chrony.conf <<EOF
pool ntp1.aliyun.com iburst
allow 192.168.100.0/24
allow 110.110.100.0/24
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/drift
rtcsync
makestep 1.0 3
EOF

# On other nodes:
cat > /etc/chrony.conf <<EOF
server 192.168.100.2 iburst
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/drift
rtcsync
makestep 1.0 3
EOF

systemctl enable --now chronyd

Ceph Package Installation

Install Ceph packages on all nodes. For simplicity, install the full suite.

yum install -y ceph-*

Cluster Initialization

On the first monitor node, create the initial cluster configuration.

cd /etc/ceph

# Generate keyrings
ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'

# Merge keyrings
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring

# Create monitor map
monmaptool --create --add ceph-node-1 110.110.100.2 --add ceph-node-2 110.110.100.3 --add ceph-node-3 110.110.100.4 --fsid $(uuidgen) /tmp/mon.map

# Set ownership
chown ceph:ceph /tmp/ceph.mon.keyring /tmp/mon.map

# Create monitor data directory
sudo -u ceph mkdir -p /var/lib/ceph/mon/ceph-ceph-node-1

# Initialize monitor
sudo -u ceph ceph-mon --cluster ceph --mkfs -i ceph-node-1 --monmap /tmp/mon.map --keyring /tmp/ceph.mon.keyring

# Start and verify
systemctl enable --now ceph-mon@ceph-node-1
ceph -s

Distribute Configuration to All Nodes

Copy critical files to other monitor nodes.

for node in ceph-node-2 ceph-node-3; do
  scp /etc/ceph/ceph.client.admin.keyring $node:/etc/ceph/
  scp /tmp/ceph.mon.keyring $node:/tmp/
  scp /tmp/mon.map $node:/tmp/
  scp /var/lib/ceph/bootstrap-osd/ceph.keyring $node:/var/lib/ceph/bootstrap-osd/
done

# On each additional monitor node:
sudo chown ceph:ceph /tmp/ceph.mon.keyring
sudo -u ceph ceph-mon --cluster ceph --mkfs -i ceph-node-2 --monmap /tmp/mon.map --keyring /tmp/ceph.mon.keyring
systemctl enable --now ceph-mon@ceph-node-2

# Repeat for ceph-node-3

Global Ceph Configuration

Create /etc/ceph/ceph.conf with the following minimal but production-ready settings:

[global]
fsid = 0bd47374-8167-4bbc-bf12-3a59d4ddd50d
mon_initial_members = ceph-node-1, ceph-node-2, ceph-node-3
mon_host = 192.168.100.2, 192.168.100.3, 192.168.100.4
cluster_network = 110.110.100.0/24
public_network = 192.168.100.0/24
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd_journal_size = 8192
osd_pool_default_size = 3
osd_pool_default_min_size = 2
osd_pool_default_pg_num = 333
osd_pool_default_pgp_num = 333
osd_crush_chooseleaf_type = 1
osd_deep_scrub_interval = 1209600
mon_osd_full_ratio = 0.90
mon_osd_nearfull_ratio = 0.80
osd_op_threads = 8
journal_aio = true
journal_dio = true
ms_nocrc = true

[mon]
mon_allow_pool_delete = true
mon_clock_drift_allowed = 2

[mgr]
mgr_modules = dashboard

[client]
rbd_cache = true
rbd_cache_size = 268435456
rbd_cache_max_dirty = 134217728
rbd_cache_max_dirty_age = 5

Deploy OSDs

On each OSD node, prepare the disks using ceph-volume. Assume each OSD uses a dedicated SSD device (e.g., /dev/nvme0n1):

ceph-volume lvm create --data /dev/nvme0n1

Verify OSD status:

ceph osd tree
ceph -s

Enable Dashboard

Activate the Ceph dashboard on a mannager node:

ceph mgr module enable dashboard
ceph dashboard create-self-signed-cert
ceph dashboard ac-user-create admin admin admin -i /dev/stdin <<EOF
admin
admin
EOF

Access the dashboard at https://<mgr-ip>:8443 using credentials admin/admin.