Cluster Installation
Refer to the official documentation for basic cluster installation:
Management and Bussiness Network Setup
Confiugre the management and business networks as described in:
Ceph Network Configuration
Create OVS bonds and VLAN interfaces for Ceph traffic:
# Configure OVS bond for Ceph
# /etc/network/interfaces
...
auto bond1
iface bond1 inet manual
ovs_bridge vmbr1
ovs_type OVSBond
ovs_bonds enp2s0f2 enp2s0f3
ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast
# Create Ceph VLAN interface
auto ceph
iface ceph inet static
address 10.10.10.101/24
ovs_type OVSIntPort
ovs_bridge vmbr1
ovs_options tag=111
# Create OVS bridge for Ceph
auto vmbr1
iface vmbr1 inet manual
ovs_type OVSBridge
ovs_ports bond1 ceph
# Final configuration example
auto lo
iface lo inet loopback
auto enp2s0f0
iface enp2s0f0 inet manual
auto enp2s0f1
iface enp2s0f1 inet manual
auto enp2s0f2
iface enp2s0f2 inet manual
auto enp2s0f3
iface enp2s0f3 inet manual
iface enp11s0 inet manual
iface enp12s0 inet manual
auto mgt
iface mgt inet static
address 192.168.0.101/24
gateway 192.168.0.254
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_options tag=110
auto ceph
iface ceph inet static
address 10.10.10.101/24
ovs_type OVSIntPort
ovs_bridge vmbr1
ovs_options tag=111
auto bond0
iface bond0 inet manual
ovs_bridge vmbr0
ovs_type OVSBond
ovs_bonds enp2s0f0 enp2s0f1
ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast
auto bond1
iface bond1 inet manual
ovs_bridge vmbr1
ovs_type OVSBond
ovs_bonds enp2s0f2 enp2s0f3
ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast
auto vmbr0
iface vmbr0 inet manual
ovs_type OVSBridge
ovs_ports bond0 mgt
auto vmbr1
iface vmbr1 inet manual
ovs_type OVSBridge
ovs_ports bond1 ceph
Huawei Switch Configuration
Configure LACP trunks on the switch side:
interface Eth-Trunk1
description link_to_pve01
port link-type trunk
undo port trunk allow-pass vlan 1
port trunk allow-pass vlan 2 to 4094
mode lacp
interface Eth-Trunk2
description link_to_pve01_ceph
port link-type access
port default vlan 111
mode lacp
interface GigabitEthernet0/0/7
description link_to_pve01
eth-trunk 1
interface GigabitEthernet0/0/8
description link_to_pve01
eth-trunk 1
interface GigabitEthernet0/0/9
description link_to_pve01_ceph
eth-trunk 2
interface GigabitEthernet0/0/10
description link_to_pve01_ceph
eth-trunk 2
Ceph Repository Setup
Add the Proxmox Ceph repositories:
# /etc/apt/sources.list.d/ceph.list
deb https://mirrors.ustc.edu.cn/proxmox/debian/ceph-quincy bookworm no-subscription
deb http://download.proxmox.com/debian/ceph-reef bookworm no-subscription
Ceph Installation
Install Ceph on all nodes:
pveceph install
Initialize Ceph Configuration
Set up the Ceph cluster network:
pveceph init --network 10.10.10.0/24
Create Monitors
On all three nodes:
pveceph mon create
Create Managers
On the remaining two nodes:
pveceph mgr create
Create OSDs
Add OSDs on each node (example with /dev/sdb and /dev/sdc):
pveceph osd create /dev/sdb
pveceph osd create /dev/sdc
Create a Pool
Create a pool for VM storage:
pveceph pool create vm --add_storages
Enable PG Autoscaler
Enable the PG autoscaler module:
ceph mgr module enable pg_autoscaler
Create CephFS
Create MDS services and a CephFS filesystem:
# On all three nodes
pveceph mds create
# Add standby replay to configuration
# /etc/pve/ceph.conf
[global]
mds standby replay = true
# Create the filesystem
pveceph fs create --pg_num 128 --add-storage
Dashboard
Access the Proxmox VE web interface to monitor the Ceph cluster.
Performance Test
Create a test pool and run benchmarks:
pveceph pool create test
# Write test
rados bench -p test 10 write --no-cleanup
# Example output:
# Bandwidth (MB/sec): 172.161
# Average IOPS: 43
# Sequential read test
rados bench -p test 10 seq
# Example output:
# Bandwidth (MB/sec): 427.097
# Average IOPS: 106