Implementing a High-Availability Redis Cluster

Understanding Redis Cluster Architecture

Redis Cluster represents a distributed implementation of Redis that provides high availability and scalability. Unlike traditional master-slave configurations, Redis Cluster operates in a decentralized manner where data is automatically partitioned across multiple nodes.

The fundamental concept behind Redis Cluster is the use of hash slots. There are 16384 hash slots distributed across all master nodes in the cluster. When a key is stored, Redis computes a hash value using the formula CRC16(key) % 16384 to determine which slot should hold the data. This slot is then mapped to a specific master node in the cluster.

Key Features of Redis Cluster

Automatic Sharding: Data is automatically partitioned across nodes
High Availability: Cluster continues operating even when some nodes fail
Master-Slave Replication: Each master node can have one or more slave nodes
Automatic Failover: Slaves can automatically replace failed masters
Online Resharding: Ability to redistribute data across nodes without service interruption
Client-side Redirects: Clients receive redirects to the correct node

Hardware Planning

For a production environment, a recommended configuration consists of 3 master nodes and 3 slave nodes, distributed across 6 separate servers to ensure proper fault tolerance.

Operating System	IP Address	Hostname	Resources	Redis Version
Kylin OS 3.3	10.0.0.31	redis-node-1	2 Cores / 2GB RAM	6.2.6
Kylin OS 3.3	10.0.0.32	redis-node-2	2 Cores / 2GB RAM	6.2.6
Kylin OS 3.3	10.0.0.33	redis-node-3	2 Cores / 2GB RAM	6.2.6
Kylin OS 3.3	10.0.0.34	redis-node-4	2 Cores / 2GB RAM	6.2.6
Kylin OS 3.3	10.0.0.35	redis-node-5	2 Cores / 2GB RAM	6.2.6
Kylin OS 3.3	10.0.0.36	redis-node-6	2 Cores / 2GB RAM	6.2.6

Cluster Deployment Process

Step 1: Download Redis

wget http://download.redis.io/releases/redis-6.2.6.tar.gz

Step 2: Install Dependencies

yum -y install gcc automake autoconf libtool make

Step 3: Configure System Settings

Disable firewall, SELinux, and unnecessary services:

# Disable firewall and other services
systemctl disable --now firewalld
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager

# Disable SELinux
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

# Verify SELinux status
grep ^SELINUX= /etc/selinux/config

Step 4: Compile and Install Redis

tar xf redis-6.2.6.tar.gz
mv redis-6.2.6 /usr/local/redis

cd /usr/local/redis
make

# Check installation status
echo $?

Step 5: Configure Environment Variables

# Add Redis to PATH
echo 'export PATH=/usr/local/redis/src:$PATH' >> /etc/profile
source /etc/profile
echo $PATH

Step 6: Create Directories

# Create data directory for Redis
mkdir -p /data/6379

Step 7: Create Configuration File

cat > /data/6379/redis.conf <<eof actual="" appendonly="" bind="" cluster-announce-ip="" cluster-config-file="" cluster-enabled="" cluster-node-timeout="" daemonize="" dbfilename="" dir="" dump.rdb="" eof="" ip="" logfile="" loglevel="" no="" node="" nodes.conf="" notice="" pidfile="" port="" protected-mode="" redissecurepassword123="" replace="" requirepass="" with="" yes=""></eof>

Step 8: Kernel Optimization

# Apply kernel optimizations
cat >>/etc/sysctl.conf<<eof apply="" changes="" eof="" net.core.netdev_max_backlog="16384" net.core.somaxconn="16384" net.ipv4.ip_local_port_range="4000" net.ipv4.route.gc_timeout="100" net.ipv4.tcp_fin_timeout="2" net.ipv4.tcp_keepalive_time="600" net.ipv4.tcp_max_orphans="16384" net.ipv4.tcp_max_syn_backlog="16384" net.ipv4.tcp_max_tw_buckets="36000" net.ipv4.tcp_syn_retries="1" net.ipv4.tcp_synack_retries="1" net.ipv4.tcp_syncookies="1" net.ipv4.tcp_tw_recycle="1" net.ipv4.tcp_tw_reuse="1" sysctl="" vm.overcommit_memory="0"></eof>

Step 9: Start Redis Service

# Start Redis with configuration
redis-server /data/6379/redis.conf

Step 10: Create Systemd Service

# Create systemd service file
cat >/usr/lib/systemd/system/redis.service<<eof after="network.target" and="" daemon-reload="" database="" description="Redis" enable="" eof="" execstart="/usr/local/redis/src/redis-server" execstop="/usr/local/redis/src/redis-cli" grep="" group="root" is="" key-value="" listening="" netstat="" persistent="" port="" redis.service="" runtimedirectory="redis" runtimedirectorymode="0755" service="" shutdown="" start="" status="" systemctl="" systemd="" the="" type="notify" user="root" verify="" wantedby="multi-user.target" wants="network-online.target"></eof>

Step 11: Handle Startup Warnings

# Address common warnings
echo '511' > /proc/sys/net/core/somaxconn
echo never > /sys/kernel/mm/transparent_hugepage/enabled
ulimit -SHn 65535
echo "* - nofile 65535" >>/etc/security/limits.conf

Forming the Cluster

Step 1: Create the Cluster

With Redis 5.0 and later, you can create a cluster using the redis-cli utility without requiring Ruby:

# Create a 3-master, 3-slave cluster
redis-cli -a RedisSecurePassword123 --cluster create \
10.0.0.31:6379 \
10.0.0.32:6379 \
10.0.0.33:6379 \
10.0.0.34:6379 \
10.0.0.35:6379 \
10.0.0.36:6379 \
--cluster-replicas 1

Troubleshooting Cluster Formation

If you encounter "Waiting for the cluster to join" errors, verify these settings:

The bind directive in redis.conf should be set to the node's actual IP address
All nodes must be able to communicate on the designated ports
Before reforming the cluster, clean previous configurations: ```

Remove cluster-specific files

rm -f nodes.conf dump.rdb appendonly.aof

Reset cluster state

redis-cli -a RedisSecurePassword123 cluster reset

Restart Redis instances

redis-server /data/6379/redis.conf

Step 2: Verify Cluster Status

# Check cluster information
redis-cli -a RedisSecurePassword123 cluster info

# List all connected nodes
redis-cli -a RedisSecurePassword123 cluster nodes | grep connected

# List master nodes
redis-cli -a RedisSecurePassword123 cluster nodes | grep master

# List slave nodes
redis-cli -a RedisSecurePassword123 cluster nodes | grep slave

Step 3: Test Cluster Functionality

Connect to any node in cluster mode (-c) and test data storage and retrieval:

# Connect to first node
redis-cli -h 10.0.0.31 -c -a RedisSecurePassword123

# Set a value
127.0.0.1:6379> set redis-cluster-test "value"
OK

# Connect to a different node
redis-cli -h 10.0.0.35 -c -a RedisSecurePassword123

# Retrieve the value - should redirect to the correct node
127.0.0.1:6379> get redis-cluster-test
-> Redirected to slot [1234] located at 10.0.0.31:6379
"value"

Cluster Management Commands

Cluster Information

# Get cluster status
redis-cli -a RedisSecurePassword123 cluster info

# List all nodes with details
redis-cli -a RedisSecurePassword123 cluster nodes

Node Management

# Add a node to the cluster
redis-cli -a RedisSecurePassword123 cluster meet <ip> <port>

# Remove a node from the cluster
redis-cli -a RedisSecurePassword123 cluster forget <node_id>

# Configure a slave to replicate from a specific master
redis-cli -a RedisSecurePassword123 cluster replicate <master_node_id>

# Save node configuration to disk
redis-cli -a RedisSecurePassword123 cluster saveconfig</master_node_id></node_id></port></ip>

Slot Management

# Assign slots to current node
redis-cli -a RedisSecurePassword123 cluster addslots <slot> [slot ...]

# Remove slots from current node
redis-cli -a RedisSecurePassword123 cluster delslots <slot> [slot ...]

# Remove all slots from current node
redis-cli -a RedisSecurePassword123 cluster flushslots

# Assign a slot to a specific node
redis-cli -a RedisSecurePassword123 cluster setslot <slot> node <node_id>

# Prepare to migrate a slot from this node
redis-cli -a RedisSecurePassword123 cluster setslot <slot> migrating <node_id>

# Prepare to receive a slot to this node
redis-cli -a RedisSecurePassword123 cluster setslot <slot> importing <node_id>

# Cancel slot migration/import
redis-cli -a RedisSecurePassword123 cluster setslot <slot> stable</slot></node_id></slot></node_id></slot></node_id></slot></slot></slot>

Key Management

# Calculate which slot a key belongs to
redis-cli -a RedisSecurePassword123 cluster keyslot <key>

# Count keys in a specific slot
redis-cli -a RedisSecurePassword123 cluster countkeysinslot <slot>

# Retrieve keys from a slot
redis-cli -a RedisSecurePassword123 cluster getkeysinslot <slot> <count></count></slot></slot></key>

Cluster Recovery Process

If the cluster becomes unstable and needs to be rebuilt:

Stop all Redis services
Delete data files (WARNING: This will result in data loss!): ```
Remove cluster configuration and data files
rm -f nodes.conf dump.rdb appendonly.aof
Restart all Redis services
Recreate the cluster using the same command as during initial setup

Master Node Failure and Recovery

Simulating Master Failure

To test failover, stop a master node:

# Stop the Redis service on a master node
systemctl stop redis.service

The cluster should automatically detect the failure and promote a slave to master. You can verify this by checking the cluster nodes:

# Check cluster status
redis-cli -a RedisSecurePassword123 cluster nodes

Recovering a Failed Master

When the original master node is brought back online, it will automatically join as a slave to the new master:

# Restart the Redis service
systemctl start redis.service

# Verify the node has joined as a slave
redis-cli -a RedisSecurePassword123 cluster nodes | grep <node_id></node_id>

The Redis logs will show the failover process:

# Sample log entries during failover:
* Marking node <node_id> as failing (quorum reached).
# Cluster state changed: fail
# Failover auth granted to <new_master_id> for epoch <epoch>
# Cluster state changed: ok
* Clear FAIL state for node <node_id>: master without slots is reachable again.</node_id></epoch></new_master_id></node_id>

Cluster Scaling Operations

Adding a New Master Node

Step 1: Prepare the New Node

# Create data directory
mkdir -p /data/6379

# Create configuration file
cat > /data/6379/redis.conf <<eof actual="" appendonly="" bind="" cluster-announce-ip="" cluster-config-file="" cluster-enabled="" cluster-node-timeout="" daemonize="" dbfilename="" dir="" dump.rdb="" eof="" ip="" logfile="" loglevel="" no="" nodes.conf="" notice="" pidfile="" port="" protected-mode="" redis="" redis-server="" redissecurepassword123="" replace="" requirepass="" service="" start="" the="" with="" yes=""></eof>

Step 2: Add Node to Cluster

# Add the new node to the cluster
redis-cli -a RedisSecurePassword123 -h 10.0.0.31 -p 6379 cluster meet 10.0.0.37 6379

# Verify the node has been added
redis-cli -a RedisSecurePassword123 -h 10.0.0.31 -p 6379 cluster nodes

Step 3: Rebalance Slots

The new node has no slots assigned. Use the resharding command to redistribute slots:

# Initiate resharding
redis-cli -a RedisSecurePassword123 --cluster reshard 10.0.0.31:6379

# Follow the prompts:
# How many slots do you want to move? (e.g., 4096 for 1/4 of total slots)
# What is the receiving node ID? (ID of the new node)
# Please enter all the source node IDs. (Type 'all' to use all nodes)
# Confirm the operation

Step 4: Verify Slot Distribution

# Check the slot distribution across all nodes
redis-cli -a RedisSecurePassword123 cluster nodes | grep -E 'master|slave'

Removing a Master Node

Step 1: Migrate Slots Away

Before removing a master node, all its slots must be migrated to other nodes:

# Identify the slots to move
redis-cli -a RedisSecurePassword123 cluster nodes | grep <node_id_to_remove>

# Use resharding to move slots
redis-cli -a RedisSecurePassword123 --cluster reshard 10.0.0.31:6379

# Follow the prompts:
# How many slots do you want to move? (number of slots on the node to remove)
# What is the receiving node ID? (ID of a node that will receive the slots)
# Please enter all the source node IDs. (ID of the node to remove)
# Confirm the operation

# Repeat for all remaining slots on the node</node_id_to_remove>

Step 2: Remove the Node

Once all slots have been migrated, remove the node from the cluster:

# Remove the node from the cluster
redis-cli -a RedisSecurePassword123 -h 10.0.0.31 -p 6379 cluster forget <node_id_to_remove></node_id_to_remove>

Step 3: Clean Up Node Data

After removing the node, clean up its data files:

# Stop the service
systemctl stop redis.service

# Remove cluster configuration files
rm -f nodes.conf dump.rdb appendonly.aof

Adding a Slave Node

Step 1: Prepare and Start the Node

Follow the same steps as when adding a master node to prepare and start the new node.

Step 2: Add Node to Cluster

# Add the new node to the cluster
redis-cli -a RedisSecurePassword123 -h 10.0.0.31 -p 6379 cluster meet 10.0.0.37 6379

Step 3: Configure as Slave

# Connect to the new node
redis-cli -a RedisSecurePassword123 -h 10.0.0.37 -p 6379

# Configure as slave of a specific master
127.0.0.1:6379> cluster replicate <master_node_id>
OK

# Exit
127.0.0.1:6379> QUIT</master_node_id>

Step 4: Verify Configuration

# Check the cluster nodes to verify the new slave
redis-cli -a RedisSecurePassword123 cluster nodes | grep slave

Removing a Slave Node

Removing a slave node is simpler than removing a master since no slot migrtaion is required:

# Remove the slave node from the cluster
redis-cli -a RedisSecurePassword123 -h 10.0.0.31 -p 6379 cluster forget <slave_node_id>

# Stop and clean up the node
systemctl stop redis.service
rm -f nodes.conf dump.rdb appendonly.aof</slave_node_id>

Tags: Redis cluster high-availability distributed-systems NoSQL

Posted on Sun, 30 Aug 2026 16:27:02 +0000 by dcooper

Freaks City

Implementing a High-Availability Redis Cluster

Implementing a High-Availability Redis Cluster

Understanding Redis Cluster Architecture

Key Features of Redis Cluster

Hardware Planning

Cluster Deployment Process

Step 1: Download Redis

Step 2: Install Dependencies

Step 3: Configure System Settings

Step 4: Compile and Install Redis

Step 5: Configure Environment Variables

Step 6: Create Directories

Step 7: Create Configuration File

Step 8: Kernel Optimization

Step 9: Start Redis Service

Step 10: Create Systemd Service

Step 11: Handle Startup Warnings

Forming the Cluster

Step 1: Create the Cluster

Troubleshooting Cluster Formation

Remove cluster-specific files

Reset cluster state

Restart Redis instances

Step 2: Verify Cluster Status

Step 3: Test Cluster Functionality

Cluster Management Commands

Cluster Information

Node Management

Slot Management

Key Management

Cluster Recovery Process

Remove cluster configuration and data files

Master Node Failure and Recovery

Simulating Master Failure

Recovering a Failed Master

Cluster Scaling Operations

Adding a New Master Node

Step 1: Prepare the New Node

Step 2: Add Node to Cluster

Step 3: Rebalance Slots

Step 4: Verify Slot Distribution

Removing a Master Node

Step 1: Migrate Slots Away

Step 2: Remove the Node

Step 3: Clean Up Node Data

Adding a Slave Node

Step 1: Prepare and Start the Node

Step 2: Add Node to Cluster

Step 3: Configure as Slave

Step 4: Verify Configuration

Removing a Slave Node

Hot Tags