Redis Sentinel is a system designed to help manage Redis instances, providing high availability through monitoring, notification, and automatic failover. It oversees master and replica instances, promoting a replica to master if the current master fails. This article explains the core concepts of Redis Sentinel and demonstrates how to integrate it into a C# application using the StackExchange.Redis library.
Redis Sentinel Overview
Sentinel is a distributed system where multiple Sentinel processes collaborate to provide high availability. Key responsibilities include:
- Monitoring: Sentinels check if your master and replica instances are working as expected by sending periodic PING commands.
- Notification: Sentinels can alert system administrators or other programs via an API when something goes wrong with monitored Redis instances.
- Automatic Failover: If a master is not working correctly, Sentinel can initiate a failover process where a replica is promoted to master, other replicas are reconfigured to use the new master, and clients are informed to connect to the new master.
- Configuration Provider: Sentinel acts as a source of authority for client service discovery: clients connect to Sentinels to ask for the address of the current master for a given service.
How Sentinel Works
Sentinel's operation involves a coordinated workflow among multiple Sentinel processes:
- Discovery and Configuration: Each Sentinel starts by reading its configuration file to know which master to monitor. It discovers connected replicas and other Sentinels by talking to the master.
- Heartbeat Monitoring: Sentinels send PING commands to all monitored instances. If a master does not reply within a configurable period (
down-after-milliseconds), the Sentinel marks it assdown(subjectively down). - Objective Down Detection: When a Sentinel marks a master as
sdown, it asks other Sentinels for their opinion. If a configurable number of Sentinels (quorum) agree, the master is marked asodown(objectively down). - Leader Election and Failover: Once
odownis confirmed, the Sentinel system holds a leader election. The leader Sentinel selects the best replica to become the new master (based on priority, replication offset, run ID, etc.) and executes the failover: it promotes the chosen replica, reconfigures other replicas to follow the new master, and updates the local configuration. - Client Notification: Sentinel updates its configurasion and can notify clients via Pub/Sub messages (like
+switch-master) or by being queried. Clients using Sentinel-aware libraries will automatically reconnect to the new master.
Configuring Redis Sentinel
Sentinel configuration is typically done in a sentinel.conf file. Below is a basic example:
# sentinel.conf
port 26379
# Monitor a master named 'mymaster' at 127.0.0.1:6379
# Require at least 2 Sentinels to agree for failover
sentinel monitor mymaster 127.0.0.1 6379 2
# Mark master as down if no response for 30000 ms
sentinel down-after-milliseconds mymaster 30000
# Timeout in ms for a failover operation
sentinel failover-timeout mymaster 180000
# If master requires password
sentinel auth-pass mymaster yourpassword
sentinel monitor: Defines the master to monitor. The final number is thequorum– the number of Sentinels that must agree that the master is down to start a failover.sentinel down-after-milliseconds: The time in milliseconds after which a subjectively down condition is triggered.sentinel failover-timeout: The maximum time allowed for a failover process.
Using Sentinel in a C# Application
StackExchange.Redis is a high-performance Redis client for .NET that fully supports Sentinel mode. When configured correctly, it automatically discovers the current master and handles failovers.
Here is a practical C# example that connects to a Redis setup protected by Sentinel:
using StackExchange.Redis;
using System;
class SentinelExample
{
static void Main(string[] args)
{
// List of Sentinel endpoints
var sentinelEndpoints = new[] { "127.0.0.1:26379", "127.0.0.2:26379", "127.0.0.3:26379" };
// Configure the connection multiplexer to use Sentinel
var config = new ConfigurationOptions
{
// Provide the service name as defined in sentinel.conf
ServiceName = "mymaster",
// Endpoints list; first one is ignored for Sentinel, but required syntactically
EndPoints = { { "127.0.0.1", 6379 } },
TieBreaker = "", // Required when using Sentinel
AllowAdmin = true // Optional: needed to execute some Sentinel commands
};
// Add Sentinel endpoints explicitly
foreach (var endpoint in sentinelEndpoints)
{
config.EndPoints.Add(endpoint);
}
// Connect using the Sentinel configuration
using (var redis = ConnectionMultiplexer.Connect(config))
{
// Subscribe to Sentinel events (e.g., master switch)
var subscriber = redis.GetSubscriber();
subscriber.Subscribe("+switch-master", (channel, message) =>
{
Console.WriteLine($"Failover occurred: {message}");
// Handle the event, e.g., refresh local caches
});
// Obtain a database reference
var db = redis.GetDatabase();
// Perform standard operations
db.StringSet("testkey", "Hello from Sentinel-enabled client!");
string value = db.StringGet("testkey");
Console.WriteLine($"Retrieved value: {value}");
}
}
}
Explanation of the code:
- Sentinel Endpoints: A list of all Sentinel servers is provided. StackExchange.Redis will connect to one of them to discover the current master and replica topology.
- Configuration: The
ServiceNameproperty must match the master name used in the Sentinel configuration (e.g.,mymaster). TheTieBreakeris set to an empty string, which is required when using Sentinel. The first endpoint inEndPointsis actually ignored by the library; the Sentinel endpoints are added separately. - Connection:
ConnectionMultiplexer.Connect(config)creates a connection that is Sentinel-aware. It automatically queries Sentinels for the current master and handles reconnections during failover. - Event Subscription: By subscribing to the
+switch-masterchannel, the application can react to failover events. For instance, it might log the event or reset persistent connections. - Database Operations: Once connected, the
GetDatabase()method works transparently regardless of which instance is currently the master.
This approach abstracts the failover complexity, allowing the application to focus on business logic while remaining resilient to master failures.