To isolate heartbeat traffic and avoid interference from production interfaces, static host routes are added for the dedicated interconnect (eth2). These persist across reboots:

# On primary node (hostname: master)
ip route add 10.20.23.111/32 dev eth2
echo "ip route add 10.20.23.111/32 dev eth2" >> /etc/rc.d/rc.local

# On secondary node (hostname: backup)
ip route add 10.20.23.115/32 dev eth2
echo "ip route add 10.20.23.115/32 dev eth2" >> /etc/rc.d/rc.local

Heartbeat Service Configuration

Heartbeat manages cluster membership, failover logic, and resource orchestration:

1. Install packages on both nodes: ``` yum install -y heartbeat heartbeat-pkg

2. Main configuration (/etc/ha.d/ha.cf): ``` debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 30 warntime 10 initdead 60 mcast eth1 225.0.0.1 694 1 0 auto_failback on node master node backup

3. Authentication key (/etc/ha.d/authkeys): ``` auth 1 1 sha1 9e8a7b2f1c6d5a0e3f8b4c1d9a7e6f0c2b1a9d8e

   
   Set strict permissions:
   
   

   chmod 600 /etc/ha.d/authkeys

4. Resource definition (/etc/ha.d/haresources): ``` master IPaddr::10.0.0.73/25/eth0

Start the service on the primary node:

systemctl start heartbeat
# Verify VIP assignment:
ip -br addr show | grep 10\.0\.0\.

Trigger manual failover to test:

heartbeat stop   # on master
# Confirm VIP appears on backup

DRBD Block-Level Replication Setup

DRBD synchronizes storage between nodes at the block level:

1. Add identical 5 GB block devices (e.g., /dev/sdb) to both servers.
2. Create GPT partition table and two partitions:

parted /dev/sdb mklabel gpt
parted /dev/sdb mkpart primary 0% 50%
parted /dev/sdb mkpart primary 50% 100%
parted /dev/sdb set 1 lvm off
parted /dev/sdb set 2 lvm off

Format the first partition and disable filesystem checks:

mkfs.ext4 /dev/sdb1
tune2fs -c 0 -i 0 /dev/sdb1

Install and load DRBD modules:

yum install -y drbd84 kmod-drbd84
modprobe drbd
echo "drbd" >> /etc/modules

Configure DRBD resource (/etc/drbd.d/data.res):

resource data {
  protocol C;
  startup {
    wfc-timeout 15;
    degr-wfc-timeout 60;
  }
  net {
    cram-hmac-alg "sha256";
    shared-secret "nfs-cluster-key";
    allow-two-primaries;
  }
  syncer {
    rate 100M;
    verify-alg crc32c;
  }
  disk {
    on-io-error detach;
  }
  on master {
    device /dev/drbd0;
    disk /dev/sdb1;
    address 10.0.0.82:7788;
    meta-disk /dev/sdb2[0];
  }
  on backup {
    device /dev/drbd0;
    disk /dev/sdb1;
    address 10.0.0.83:7788;
    meta-disk /dev/sdb2[0];
  }
}

Initialize metadata and bring up the resource:

drbdadm create-md data
drbdadm up data
# Promote one node to Primary:
drbdadm --force primary data

Verify status:

cat /proc/drbd
drbdadm status

Mount the replicated device:

mkdir -p /data
mount /dev/drbd0 /data

NFS Server Deployment

Export the DRBD-mounted filesystem via NFS:

1. Install required packages:

yum install -y nfs-utils rpcbind

Define export policy (/etc/exports):

/data 10.0.0.0/24(rw,sync,no_root_squash,fsid=0)

Enable and start services (managed by Heartbeat, not systemd):

systemctl enable rpcbind
systemctl start rpcbind
# Do NOT enable nfs-server — it will be controlled externally

Orchestration Integration

Update Heartbeat’s resource list to coordinate DRBD, filesystem, and NFS liefcycle:

# In /etc/ha.d/haresources:
master \
  drbddisk::data \
  Filesystem::/dev/drbd0::/data::ext4 \
  nfsd \
  IPaddr::10.0.0.73/25/eth0

Create a custom NFS resource script (/etc/ha.d/resource.d/nfsd):

#!/bin/bash
NFS_SERVICES="rpcbind nfs-server rpc-statd rpc-mountd rpc-rquotad"

case "$1" in
  start)
    systemctl start $NFS_SERVICES 2>/dev/null || true
    exportfs -ra
    ;;
  stop)
    exportfs -u -a
    systemctl stop $NFS_SERVICES 2>/dev/null || true
    ;;
  status)
    systemctl is-active --quiet nfs-server && echo "running" || echo "stopped"
    ;;
  *)
    echo "Usage: $0 {start|stop|status}"
    exit 1
    ;;
esac

Make executable and test full stack failover:

chmod +x /etc/ha.d/resource.d/nfsd
# Restart Heartbeat on primary to reload config
systemctl restart heartbeat

Validate client accessibility after failover using standard NFS tools (showmount, mount.nfs).