Core Concepts and System Architecture

Redis Sentinel provides high availability for Redis deployments by managing automatic failover for master nodes. Built on top of the standard master-slave replication mechanism, it ensures continuous service availability when the primary node fails.

Primary Functions

• Monitoring: Continuous health checks on both master and replica nodes to detect failures.
• Automatic Failover: When the master node becomes unavailable, Sentinel promotes a replica to master and reconfigures other replicas accordingly.
• Configuration Provider: Clients query Sentinel for the current master address, eliminating hardcoded connection details.
• Notification: Sentinel broadcasts failover events to connected clients through pub/sub channels.

Component Structure

The architecture consists of two distinct node types:

1. Sentinel Nodes: Special Redis processes that monitor the system. They do not store application data.
2. Data Nodes: Standard Redis instances holding the actual data (master and replicas).

Sentinel operates as a distributed system where multiple independent nodes collectively judge the state of the master. This quorum-based approach prevents false positives and ensures the system remains robust even if individual Sentinel nodes fail.

Deployment Configuration

The following example deploys a complete Sentinel setup on a single server (192.168.1.100) using distinct ports for isolation.

Master-Replica Setup

Configure one master (port 6379) and two replicas (ports 6380, 6381). Connect replicas to the master:

# On replica nodes (6380 and 6381)
replicaof 192.168.1.100 6379

Verify the replication status on the master:

INFO replication

Sentinel Node Configuration

Deploy three Sentinel instances (ports 26379, 26380, 26381). Create a configuration file for each, such as sentinel-26379.conf:

port 26379
sentinel monitor primary-service 192.168.1.100 6379 2
sentinel down-after-milliseconds primary-service 30000
sentinel failover-timeout primary-service 180000
sentinel parallel-syncs primary-service 1

The sentinel monitor directive defines the monitored master. The value 2 represents the quorum—minimum Sentinels required to agree on a failover decision.

Start each Sentinel process:

redis-server sentinel-26379.conf --sentinel
# or
redis-sentinel sentinel-26379.conf

Upon startup, Sentinels automatically discover the master, all replicas, and peer Sentinel nodes through internal communication.

Client Integration

Clients interact with Sentinel to obtain the current master address, decoupling application code from specific server instances.

Java Implementation Example

import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisSentinelPool;
import java.util.Set;
import java.util.HashSet;

public class SentinelClient {
    public static void main(String[] args) {
        String masterAlias = "primary-service";
        Set<String> sentinelAddresses = new HashSet<>();
        sentinelAddresses.add("192.168.1.100:26379");
        sentinelAddresses.add("192.168.1.100:26380");
        sentinelAddresses.add("192.168.1.100:26381");
        
        try (JedisSentinelPool pool = new JedisSentinelPool(masterAlias, sentinelAddresses)) {
            try (Jedis connection = pool.getResource()) {
                connection.set("user:token", "abc123xyz");
                System.out.println(connection.get("user:token"));
            }
        }
    }
}

Connection Pool Mechanics

The JedisSentinelPool constructor iterates through the provided Sentinel addresses, querying each for the current master location using the SENTINEL get-master-addr-by-name command. Once found, it establishes the connection pool.

Additionally, the pool spawns listener threads for each Sentinel, subscribing to the +switch-master channel. When a failover occurs, the listener triggers a pool reinitialization with the new master address.

// Simplified listener logic
public void onSwitchMaster(String message) {
    String[] parts = message.split(" ");
    if (targetMasterName.equals(parts[0])) {
        String newHost = parts[3];
        int newPort = Integer.parseInt(parts[4]);
        reinitializePool(newHost, newPort);
    }
}

Internal Implementation Mechanisms

Three Periodic Tasks

Sentinel relies on three scheduled tasks for monitoring and discovery:

1. INFO Command (10-second interval): Sent to the master to discover replicas. This explains how Sentinels automatically detect new replicas without explicit configuration. The command also updates topology information during or after failover.
2. Pub/Sub Announcement (2-second interval): Each Sentinel publishes its presence and view of the master to the __sentinel__:hello channel on the master. Sentinels subscribe to this channel, enabling mutual discovery and state synchronization.
3. PING Health Check (1-second interval): Sent to all instances (master, replicas, and peer Sentinels) to determine reachability. This data feeds into the failure detection logic.

Failure Detection: Subjective and Objective Offline

The down-after-milliseconds parameter defines the timeout threshold.

Subjective Offline (S_DOWN): If a Sentinel receives no valid PING response from a node within the configured timeout, it marks that node as subjectively down. This represents a single Sentinel's opinion and may be a false positive.

Objective Offline (O_DOWN): For the master node, after marking it subjectively down, a Sentinel queries peers using SENTINEL is-master-down-by-addr. If the number of affirmatives reaches the configured quorum, the master is marked objectively down. Only the master undergoes this collective verification; replicas are subjectively assessed only.

Leader Sentinel Election

Once the master is objectively down, a single Sentinel must lead the failover. The election uses a consensus protocol similar to Raft:

1. Any Sentinel that detects objective offline status requests leadership from peers via SENTINEL is-master-down-by-addr, including its own run ID as a candidate.
2. Peers grant approval only to the first request received (one vote per election cycle).
3. A candidate wins if it obtains max(quorum, sentinels/2 + 1) votes.
4. If no majority emerges, the election restarts after a delay.

This mechanism ensures exactly one Sentinel orchestrates the failover, preventing conflicting recovery actions.

Failover Execution

The elected leader Sentinel performs the following steps:

1. Select New Master: Filter out unhealthy replicas (subjectively down or disconnected). From remaining candidates, prioritize by:
   • Lowest replica-priority value (configurable weight).
   • Greatest replication offset (most data synchronized).
   • Smallest run ID (tiebreaker for deterministic selection).
2. Promote Selected Replica: Send REPLICAOF NO ONE to the chosen node, converting it to a master.
3. Reconfigure Other Replicas: Command remaining replicas to follow the new master via REPLICAOF.
4. Update Old Master Configuration: The failed master is configured as a replica of the new master, ensuring it rejoins correctly upon recovery.