Sentinel Overview

Sentinel is a distributed system designed to monitor master-slave Redis deployments. When a master becomes unavailable, Sentinel automatically promotes a slave to master and redirects all other slaves to the new master.

Core Capabilities

Monitoring

Continuously checks master and slave availability
Detects master存活检测 and replication status

Notification

Alerts other sentinels and client applications when issues occur
Provides real-time status updates through pub/sub channels

Automatic Failover

Promotes a suitable slave to master
Reconfigures remaining slaves to follow the new master
Notifies clients of the new master address

Sentinel deployments require an odd number of instances (typically 3 or 5) and do not serve data.

Configuration

The sentinel.conf file controls Sentinel behavior:

cat sentinel.conf | grep -v '#' | grep -v "^$"

port 26379
dir /tmp
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000

sentinel monitor

Defines monitoring target. The quorum parameter specifies how many sentinels must agree that a master is failed before initiating failover. Setting quorum to 2 means at least two sentinels must confirm the master is down before triggering failover. Generally, set this to (sentinel_count / 2) + 1.

sentinel down-after-milliseconds

Configures the timeout for ping responses. If no pong is received within the specified milliseconds, the instance is marked as subjective down.

sentinel parallel-syncs

Controls how many slaves can simultaneously reconfigure to the new master during failover. Limiting this prevents overwhelming the new master with replication requests, which could saturate network bandwidth and disk I/O.

sentinel failover-timeout

Maximum duration for failover operations. If exceeded, the failover is considered failed.

sentinel auth-pass

Required when masters have authentication enabled.

Starting Sentinel

redis-sentinel sentinel-端口号.conf

Working Principles

System Architecture

A Sentinel deployment consists of one or more Sentinel instances running in special mode. Each Sentinel can:

1. Monitor any number of masters and their slaves
2. Automatically promote a slave when the master fails

Note: Sentinels cannot execute transactional commands, Lua scripts, or database commands.

Phase 1: Connection Establishment

Step 1: Initialize Sentinel

The Sentinel initializes its internal state and establishes network connections to monitored masters.

Process:

1. Initialize server
2. Replace standard Redis server code with Sentinel-specific code
3. Initialize sentinelState structure (defined in sentinel.c)
4. Parse configuration file to populate the masters dictionary
5. Create two async connections per master: command connection and subscription connection

The command connection sends commands and receives replies. The subscription connection subscribes to the __sentinel__:hello channel for discovering other sentinels.

sentinelState structure:

struct sentinelState {
    char myid[CONFIG_RUN_ID_SIZE+1];
    uint64_t current_epoch;
    dict *masters;
    int tilt;
    int running_scripts;
    mstime_t tilt_start_time;
    mstime_t previous_time;
    list *scripts_queue;
    char *announce_ip;
    int announce_port;
    unsigned long simfailure_flags;
    int deny_scripts_reconfig;
} sentinel;

The masters dictionary stores information about all monitored masters, keyed by the master name from configuration. Each value is a sentinelRedisInstance structure.

sentinelRedisInstance structure:

typedef struct sentinelRedisInstance {
    int flags;
    char *name;
    char *runid;
    uint64_t config_epoch;
    sentinelAddr *addr;
    instanceLink *link;
    mstime_t last_pub_time;
    mstime_t last_hello_time;
    mstime_t last_master_down_reply_time;
    mstime_t s_down_since_time;
    mstime_t o_down_since_time;
    mstime_t down_after_period;
    mstime_t info_refresh;
    int role_reported;
    mstime_t role_reported_time;
    mstime_t slave_conf_change_time;
    dict *sentinels;
    dict *slaves;
    unsigned int quorum;
    int parallel_syncs;
    char *auth_pass;
    mstime_t master_link_down_time;
    int slave_priority;
    mstime_t slave_reconf_sent_time;
    struct sentinelRedisInstance *master;
    char *slave_master_host;
    int slave_master_port;
    int slave_master_link_status;
    unsigned long long slave_repl_offset;
    char *leader;
    uint64_t leader_epoch;
    uint64_t failover_epoch;
    int failover_state;
    mstime_t failover_state_change_time;
    mstime_t failover_start_time;
    mstime_t failover_timeout;
    mstime_t failover_delay_logged;
    struct sentinelRedisInstance *promoted_slave;
    char *notification_script;
    char *client_reconfig_script;
    sds info;
} sentinelRedisInstance;

Step 2: Fetch Master Information

After initialization, Sentinel sends INFO commands every 10 seconds through the command connection to retrieve master details and slave list. Slave information is stored in the slaves dictionary, keyed by ip:port.

Step 3: Connect to Slaves

Using information obtained from the master, Sentinel establishes command and subscription connections to each slave. INFO commands fetch detailed slave information.

Step 4: Pub/Sub Channel Communication

Sentinel publishes its own information to the __sentinel__:hello channel every 2 seconds. All sentinels monitoring the same master subscribe to this channel to discover each other and exchange configuration updates.

Connection establishment sequence:

Sentinel starts =>
Initialize and connect to all masters (command + subscription) =>
Retrieve master info via INFO command =>
Connect to all slaves similarly =>
Subscribe channels established =>
Publish hello messages containing current configuration =>
Other sentinels receive messages, compare versions, and update if newer =>
Command connections established between all sentinels monitoring the same master

Phase 2: Monitoring and Notification

Sentinel continuously monitors masters, slaves, and other sentinels through periodic commands:

• PING commands (every second) to check availability
• INFO commands to gather detailed instance information
• PUBLISH commands to broadcast current configuration

How notification works:

When Sentinel A publishes to the __sentinel__:hello channel, all other sentinels subscribed to that channel receive the message. This message contains Sentinel A's configuration and the monitored master's configuration. Other sentinels compare their local configuration with the received one and update if the incoming version is newer.

This pub/sub mechanism enables automatic discovery without manual configuration of other sentinel addresses.

Phase 3: Failover

Instance States (flags):

#define SRI_MASTER  (1<<0)
#define SRI_SLAVE   (1<<1)
#define SRI_SENTINEL (1<<2)
#define SRI_S_DOWN (1<<3)
#define SRI_O_DOWN (1<<4)
#define SRI_MASTER_DOWN (1<<5)
#define SRI_FAILOVER_IN_PROGRESS (1<<6)
#define SRI_PROMOTED (1<<7)
#define SRI_RECONF_SENT (1<<8)
#define SRI_RECONF_INPROG (1<<9)
#define SRI_RECONF_DONE (1<<10)
#define SRI_FORCE_FAILOVER (1<<11)
#define SRI_SCRIPT_KILL_SENT (1<<12)

Step 1: Subjective Down Detection

Sentinel sends PING commands every second to all connected instances. If no valid reply is received within the down-after-milliseconds threshold, the instance is marked as SRI_S_DOWN (subjectively down).

Step 2: Objective Down Detection

When enough sentinels (quorum value from configuration) mark the same master as subjective down, the master transitions to SRI_O_DOWN (objectively down).

Step 3: Leader Election

Sentinels that voted for the master to be objectively down participate in leader election. The sentinel with the highest priority or earliest run ID becomes the leader responsible for orchestrating failover.

Step 4: Execute Failover

The elected leader:

1. Selects the best slave based on priority, replication offset, and connectivity
2. Promotes the selected slave to master
3. Configures remaining slaves to replicate from the new master
4. Updates configuration across all sentinels

Operational Summary

Sentinel Activities:

Phase	Actions
Connection Setup	Establish connections to masters/slaves, discover other sentinels via pub/sub
Monitoring	Send PING commands, receive responses, detect failures
Notification	Publish configuration updates through channels, receive and validate updates from peers
Failover	Detect failures, elect leader, promote slave, reconfigure remaining slaves

Commands Sent:

• INFO: Retrieve master/slave details
• PING: Check instance availability
• PUBLISH: Broadcast sentinel information

Information Received:

• Replies to INFO/PING commands
• Pub/sub messages from other sentinels containing configuration and presence updates