Background of Replication

Server Requirements

Internet applications demand high concurrency, performance, and availability. Availability is calculated as:

Availability = (Total Yearly Time - Downtime) / Total Yearly Time

A standard target is 99.999% (five nines), allowing for roughly 5.25 minutes of downtime per year.

Limitations of a Single Instance

A standalone Redis instance faces risks such as hardware failure and memory exhaustion. To mitigate single points of failure, data must be replicated across multiple interconnected servers. This ensures that if one node fails, others continue to serve requests, achieving high availability and data redundancy.

Multi-Server Strategy

• Data Source (Master): Handles write operations (create, update, delete).
• Data Recipient (Slave): Handles read operations. Writes are typically blocked.

The core challenge is maintaining data consistency between the master and its replicas.

Definition of Replication

Replication allows one Redis server to duplicate another. The source is the master, and the destination is the slave. This mechanism is often used to reduce load on the primary node.

Characteristics of Master and Slave

• Master: Accepts writes; propagates changes automatically to slaves.
• Slave: Read-only by default.

High Availability Structures

• Slave Failure: Remaining slaves redistribute the read load.
• Master Failure: A slave can be promoted to master (manual or via Sentinel).
• Master Overload: Use a hierarchical structure (slave acting as master to other slaves) or implement a cluster of multiple masters.

Core Concepts of Redis Replication

Objectives of Replication

1. Read/Write Scaling: Isolates writes on master and reads on slaves (Read/Write Separation).
2. Load Balancing: Distributes read traffic across multiple slaves to increase throughput.
3. Fault Tolerance: Provides rapid recovery if the master fails.
4. Data Redundancy: Acts as a hot backup distinct from persistence (RDB/AOF).
5. Foundation for HA: Essential for building Sentinel or Cluster modes.

The Replication Workflow

The process is divided into three distinct phases:

1. Connection Establishment
2. Data Synchronization (Full or Partial)
3. Command Propagation

Phase	Data Consistency Status	Trigger Condition
Connection Establishment	Not synced	Initial setup
Full Sync	Large data discrepancy	First connection or long disconnect
Partial Sync	Minor data discrepancy	Reconnection after short interruption
Command Propagation	Near real-time sync	Stable connection post-sync

Phase 1: Connection Establishment

The SLAVEOF command configures a server as a replica of another.

SLAVEOF <master_ip> <master_port>

To unset a replica and make it a master again:

SLAVEOF NO ONE

Workflow:

1. Slave saves the master's IP and port.
2. Slave initiates a socket connection to the master.
3. Slave sends a PING to verify the link.
4. If configured, authentication occurs.
5. Master records the slave's port for future communication.

Configuration Example:

To enable replication via configuration file (redis-slave.conf):

port 6380
slaveof 127.0.0.1 6379
daemonize no
appendonly yes

Verification: Use the INFO replication command on the master to see connected slaves.

127.0.0.1:6379> INFO replication
# Replication
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=6380,state=online,offset=98,lag=1

Phase 2: Data Synchronization

Modern Redis (2.8+) uses the PSYNC command instead of the legacy SYNC.

Workflow:

1. Slave requests sync using PSYNC.
2. If codnitions allow (matching Run ID and valid Offset), Master performs a Partial Sync.
3. Otherwise, Master performs a Full Sync:
   • Master executes BGSAVE to create an RDB snapshot.
   • Master stores new write commands in a Replication Backlog Buffer.
   • RDB file is sent to Slave.
   • Slave flushes old data and loads the RDB.
   • Master sends the buffered commands (AOF-style) to Slave.

Considerations:

• Master Side: Avoid full syncs during peak traffic. Ensure repl-backlog-size is large enough to prevent buffer overflow, wich forces another full sync.
• Slave Side: Disable read services during full sync if strict consistency is required. For multiple slaves, consider a tree topology to reduce master bandwidth usage.

Phase 3: Command Propagation

Once synchronized, the master streams every write command to the slaves to maintain consistency.

• Stable Connection: Commands are forwarded immediately.
• Short Disconnect: Handled via Partial Sync (using backlog buffer).
• Long Disconnect: Triggers a new Full Sync.

Partial Synchronization Mechanism

Partial sync is efficient because it only transfers the data missed during a disconnect.

Core Components

1. Run ID (Unique Identifier): A 40-character hex string identifying a specific server instance. Generated at startup.
2. Replication Backlog Buffer: A fixed-size FIFO queue (default 1MB) on the master storing recent write commands.
3. Replication Offset: A marker indicating the position in the stream/buffer.

Run ID

When a slave connects for the first time, the master sends its Run ID. The slave stores this. If the slave reconnects and the Run ID doesn't match (e.g., master restarted), a full sync is triggered.

Check Run ID using:

127.0.0.1:6379> INFO server
... 
run_id:6c4a123b456... 

Replication Offset

• Master Offset: Tracks total bytes sent to replicas.
• Slave Offset: Tracks total bytes processed from the master.

If Slave Offset < Master Offset, the slave is lagging. The master uses the backlog buffer to fill the gap.

Heartbeat Mechanism

To ensure the link is alive, both sides exchange heartbeat signals.

• Master Heartbeat:

  • Command: PING
  • Frequency: Defined by repl-ping-slave-period (default 10s).
  • Purpose: Check if slave is responsive.

• Slave Heartbeat:

  • Command: REPLCONF ACK <offset>
  • Frequency: Every 1 second.
  • Purpose: Report current offset to master and check master liveness.

Note: If min-slaves-to-write and min-slaves-max-lag are configured, the master will stop accepting writes if the required number of slaves are not available or are lagging too far.

Common Issues and Solutions

Frequent Full Resynchronization

Scenario A: Master Restart. When a master restarts, its Run ID changes, causing all slaves to perform a full sync.

Solution: Redis 4.0+ attempts to persist the Run ID and Offset in the RDB file during shutdown (shutdown save). Upon restart, it reloads these values, preventing unnecessary full syncs.

Scenario B: Small Backlog Buffer. If the network disconnects for longer than it takes to overflow the 1MB backlog buffer, the offset becomes invalid, forcing a full sync.

Solution: Increase repl-backlog-size. Calculate: 2 * (Average Reconnect Time) * (Write Bytes Per Second).

Frequent Network Timeouts

Issue: Slaves executing heavy commands (like KEYS *) may lag and fail to respond to heartbeats.

Solution: Adjust repl-timeout (default 60s). Ensure this value is significantly larger (5-10x) than repl-ping-slave-period to avoid false positives.

Data Inconsistency Across Slaves

Cause: Network latency or heavy load on specific slaves.

Solution:

1. Deploy master and slaves in the same network/data center.
2. Monitor offsets. If a slave lags significantly, you can temporarily stop serving it reads:

slave-serve-stale-data no

Summary

The goal of replication is data consistency. The lifecycle is:

1. Link: Slave connects to Master via SLAVEOF.
2. Sync: Large discrepancies are handled via Full Sync (RDB). Smaller gaps use Partial Sync (Buffer).
3. Propagate: Steady-state where Master streams commands to Slaves.

The system relies on the Run ID to identify the master and the Offset within the Backlog Buffer to determine what data needs to be sent.