To configure a highly available SaltStack environment, you must set up a multi-master architecture where minions can communicate with multiple master servers. Additionally, configuration files, state files, and keys must be synchronized between the primary and secondary masters to ensure consistency during a failover.

1. Configuring Salt Minions

On all managed minion nodes, update the configuration file to point to both master IP addresses. Edit the minion configuration file located at /etc/salt/minion:

master:
  - 10.0.0.10
  - 10.0.0.11

2. Setting Up Data Synchronization on the Primary Master

On the active (primary) master, install and configure rsync to facilitate the transfer of Salt directories to the backup master.

Install the rsync daemon:

yum install -y rsync

Create or modify the /etc/rsyncd.conf configuration file. This defines the modules for the Salt states, PKI keys, and master configurasions. Ensure you restrict access to the backup master's IP address.

uid = root
gid = root
use chroot = no
max connections = 200
timeout = 600
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log

[salt-states]
path = /srv/salt
ignore errors
read only = false
list = false
auth users = salt_sync
hosts allow = 10.0.0.11
secrets file = /etc/rsync/rsyncd.secrets

[salt-pki]
path = /etc/salt/pki/master
ignore errors
read only = false
list = false
auth users = salt_sync
hosts allow = 10.0.0.11
secrets file = /etc/rsync/rsyncd.secrets

[salt-master-conf]
path = /etc/salt/master.d
ignore errors
read only = false
list = false
auth users = salt_sync
hosts allow = 10.0.0.11
secrets file = /etc/rsync/rsyncd.secrets

[salt-root-conf]
path = /etc/salt
ignore errors
read only = false
list = false
auth users = salt_sync
hosts allow = 10.0.0.11
secrets file = /etc/rsync/rsyncd.secrets

Create the directory for the credantials file and define the user and password:

mkdir -p /etc/rsync
echo "salt_sync:SecureP@ssw0rd" > /etc/rsync/rsyncd.secrets
chmod 600 /etc/rsync/rsyncd.secrets

Start the rsync daemon:

rsync --daemon --config=/etc/rsyncd.conf

3. Configuring the Backup (Passive) Master

On the secondary master server, install rsync and configure the credentials to pull data from the primary master.

yum install -y rsync
mkdir -p /etc/rsync
echo "SecureP@ssw0rd" > /etc/rsync/rsync_client.secrets
chmod 600 /etc/rsync/rsync_client.secrets

Create a synchronization script, such as /usr/local/bin/pull_salt_data.sh, to copy the necessary directories:

#!/bin/bash
# Sync Salt States
rsync -auvz --progress --password-file=/etc/rsync/rsync_client.secrets salt_sync@10.0.0.10::salt-states /srv/salt

# Sync Master PKI Keys
rsync -auvz --progress --password-file=/etc/rsync/rsync_client.secrets salt_sync@10.0.0.10::salt-pki /etc/salt/pki/master

# Sync Master Configuration Directory
rsync -auvz --progress --password-file=/etc/rsync/rsync_client.secrets salt_sync@10.0.0.10::salt-master-conf /etc/salt/master.d/

# Sync Main Salt Config
rsync -auvz --progress --password-file=/etc/rsync/rsync_client.secrets salt_sync@10.0.0.10::salt-root-conf/master /etc/salt/

Make the script executable and schedule it to run every 3 minutes via crontab to ensure near real-time consistency. Note that if configurattion changes occur on the primary master, the Salt Master service on the backup server should be restarted after synchronization.

chmod +x /usr/local/bin/pull_salt_data.sh
echo "*/3 * * * * /usr/bin/bash /usr/local/bin/pull_salt_data.sh" | crontab -

4. Restarting Services

Apply the configuration changes by restarting the Salt Minion service on all managed nodes:

systemctl restart salt-minion

Finally, restart the Salt Master service on both the primary and secondary servers. If your environment uses an external job cache (e.g., MySQL), ensure the database permissions are updated to allow connections from the new master IP before restarting.

systemctl restart salt-master