Data Types and Internal Encodings

Redis distinguishes between external data types exposed to users and internal encoding mechanisms that optimize memory usage. Understanding these mappings helps in capacity planning and performance tuning.

String Implementation

Strings utilize Simple Dynamic Strings (SDS) rather than raw C strings. SDS maintains metadata including length and available capacity, enabling O(1) length retrieval and preventing buffer overflows during concatenation operations. Unlike conventional null-terminated strings, SDS supports binary-safe storage, allowing images or serialized data to be stored directly.

List Implementation (Quicklist)

Prior to version 3.2, lists utilized dual-linked lists or ziplists depending on size thresholds. Modern Redis employs quicklists exclusively—a hybrid structure combining linked lists with compressed ziplist nodes. Each node stores multiple elements in contiguous memory, reducing pointer overhead while maintaining efficient insertion capabilities at both ends.

Hash Internals

Small hashes (fewer than 512 entries with values under 64 bytes) store data in listpack structures (replacing ziplists in version 7.0), while larger datasets transition to standard hash tables. This dual approach optimizes memory for small objects while maintaining O(1) access for large datasets.

Set Internals

Sets containing only integers (within 64-bit signed range) and fewer than 512 elements use intsets—sorted integer arrays that support binary search. Exceeding either constraint triggers conversion to hash table storage.

Sorted Set Internals

ZSets employ skiplists paired with hash tables. The hash table provides O(1) score lookups by member, while the skiplist enables O(log N) range queries and ordered traversal. Small collections use listpack encoding instead of skiplists.

Specialized Data Types

Bitmaps: Store bit-level states efficiently for attendance tracking or feature flags. A single string can represent 2^32 bits (512MB).

HyperLogLog: Probabilistic cardinality estimation structure using 12KB maximum memory, ideal for unique visitor counting with 0.81% standard error.

Geospatial: Sorted set variant storing longitude/latitude pairs with geohash encoding, supporting radius queries and distance calculations.

Streams: Log data structure with consumer group support, offering message persistence and acknowledgment tracking superior to list-based queues.

Threading Architecture

While often described as single-threaded, Redis utilizes an event-driven architecture with multiple supporting threads.

The primary thread executes an event loop utilizing epoll/kqueue for I/O multiplexing. It handles connection acceptance, protocol parsing, command execution, and response serialization sequentially. This design eliminates context switching overhead and race conditions, crucial given that most operations are memory-bound rather than CPU-bound.

Background I/O (BIO) threads handle three specific operations asynchronously:

• File closure: Delayed file descriptor closing after AOF rewrites
• AOF synchronization: Fsync operations when appendfsync everysec is configured
• Lazy memory reclamation: Asynchronous deletion of large objects via UNLINK or FLUSHDB ASYNC

Version 6.0 introduced optional I/O threading for network operations. When enabled via io-threads, multiple threads handle socket reading and writing, while command execution remains single-threaded. Configuration recommendations suggest N-1 threads for N CPU cores (e.g., 3 threads for 4-core systems).

Durability Mechanisms

AOF (Append-Only File)

AOF logs write commands in a human-readable format. Three synchronization policies control durability versus performance:

• always: Synchronous disk writes on every command (maximum safety, lowest throughput)
• everysec: Buffer writes with one-second fsync intervals (default, balanced approach)
• no: Operating system controlled flushing (highest throughput, risk of data loss)

As AOF grows, rewrite operations compress it by creating a minimal command set representing current state. This utilizes copy-on-write (COW) forking—child processes inherit memory pages read-only, with modifications triggering page copies, allowing non-blocking rewrites. During rewriting, new commands append to both the existing AOF and an in-memory rewrite buffer, merged post-completion.

RDB (Redis Database)

RDB creates point-in-time snapshots of memory state. The BGSAVE command forks a child process to write the binary dump, avoiding blocking the main thread. Automated snapshots trigger based on modification counts within time windows (e.g., 900 seconds with 1+ changes).

Unlike AOF's command logs, RDB stores raw data values, enabling faster restart loading but potentially greater data loss between snapshots.

Hybrid Persistence (4.0+)

Combining both approaches, hybrid mode writes RDB-formatted full data at the beginning of AOF files, followed by incremental AOF commands. This accelerates restarts (RDB speed) while maintaining durability (AOF granularity).

Distributed Architecture

Replication

Master-replica topology enables read scaling and failover protection. Replicas asynchronously consume the replication stream—either full RDB synchronization followed by command propagation, or partial resynchronization (PSYNC) using backlog buffers for brief disconnections.

By default, replicas ignore expiration commands from clients; the master handles TTL expiration and propagates DEL operations to maintain consistency.

Sentinel

Sentinel provides automatic failover without manual intervention. Multiple Sentinel processes monitor master health via heartbeat checks. Upon master failure detection, they execute leader election among themselves, then promote the most current replica to master status and reconfigure remaining nodes.

Cluster Mode

Redis Cluster partitions data across 16,384 hash slots using CRC16(key) % 16384 mapping. Nodes own specific slot ranges, with configurations supporting:

• Automatic allocation: Even distribution during cluster creation
• Manual assignment: Precise control via CLUSTER ADDSLOTS

Cluster topology requires complete slot coverage (all 16384 slots assigned) to accept write operations. Client libraries cache slot-to-node mappings and handle MOVED/ASK redirections when topology changes.

Split-Brain Mitigation

To prevent dual-master scenarios during network partitions, configure:

min-replicas-to-write 2
min-replicas-max-lag 10

These settings reject writes unless at least 2 replicas acknowledge data within 10 seconds, ensuring majority consensus before accepting mutations.

Memory Management

Expiration Strategies

Redis employs probabilistic expiration rather than precise timers:

Lazy Expiration: Checks TTL on access, deleting expired keys during read operations. This minimizes CPU overhead but risks memory retention of never-accessed keys.

Active Expiration: Background processes sample 20 random keys per second, deleting expired entries. If >25% of sampled keys expire, the process repeats immediately (capped at 25ms execution to prevent blocking).

Eviction Policies

When memory exceeds maxmemory, Redis selects keys for removal based on eight algorithms:

No Eviction: Returns errors for write commands (default in 3.0+).

Volatile TTL: Evicts shortest TTL keys among those with expiration.

Volatile Random: Random selection from expired-key set.

Allkeys Random: Random selection from entire dataset.

Volatile LRU: Least recently used among keys with TTL (approximated via sampling).

Allkeys LRU: Least recently used across all keys.

Volatile LFU: Least frequently used among expiring keys (4.0+).

Allkeys LFU: Least frequently used across entire dataset.

LRU implementation uses 24-bit fields storing last access timestamps, sampling 5 random keys to approximate true LRU without maintaining expensive linked lists. LFU divides the same field into 16-bit last decrement time and 8-bit logarithmic counter for access frequency tracking.

Caching Patterns and Anti-Patterns

Cache Avalanche Prevention

When massive key expiration occurs simultaneously, database load spikes. Mitigation strategies include:

• Jittered TTL: Add random offsets (0-300 seconds) to expiration times
• Preemptive Refresh: Background processes update hot data before expiration

Cache Breakdown Protection

For singular hot keys expiring under high concurrency:

• Mutex Locking: Use SETNX to gate database access, allowing single reconstruction
• Logical TTL: Maintain permanent cache with embedded expiration timestamps, refreshed asynchronously

Cache Penetration Defense

Requests for non-existent data bypassing cache:

• Null Object Pattern: Cache negative results with short TTL
• Bloom Filters: Probabilistic data structures checking key existence before database queries
• Request Validation: Input sanitization to prevent invalid key generation

Update Strategies

Cache-Aside: Application manages cache explicitly—reading through on misses, invalidating on writes (not updating). This favors eventual consistency and simplifies conflict resolution.

Write-Through: Cache acts as proxy, synchronously updating backing stores. Rarely used with Redis due to lack of built-in database integration.

Write-Behind: Async batch updates to database. Not natively supported but implementable via streams or external processors.

Operational Practices

Handling Large Keys

Objects exceeding 10KB (strings) or 5000 elements (collections) cause latency spikes and network congestion.

Detection Methods:

redis-cli --bigkeys

For programmatic detection:

import redis

def scan_large_keys(redis_client, threshold_bytes=10240):
    cursor = 0
    large_keys = []
    while True:
        cursor, keys = redis_client.scan(cursor, count=100)
        for key in keys:
            size = redis_client.memory_usage(key)
            if size > threshold_bytes:
                large_keys.append((key, size))
        if cursor == 0:
            break
    return large_keys

Safe Deletion: For Redis 4.0+, use UNLINK for asynchronous deletion. For earlier versions, batch deletion prevents blocking:

def prune_large_hash(conn, key_name, chunk_size=100):
    scan_cursor = 0
    while True:
        scan_cursor, fields = conn.hscan(key_name, cursor=scan_cursor, count=chunk_size)
        if fields:
            conn.hdel(key_name, *fields.keys())
        if scan_cursor == 0:
            break

def truncate_large_list(conn, key_name, batch=100):
    while conn.llen(key_name) > 0:
        conn.ltrim(key_name, 0, -(batch + 1))

def prune_large_set(conn, key_name, chunk_size=100):
    cursor = 0
    while True:
        cursor, members = conn.sscan(key_name, cursor=cursor, count=chunk_size)
        if members:
            conn.srem(key_name, *members)
        if cursor == 0:
            break

def truncate_sorted_set(conn, key_name, batch=100):
    while conn.zcard(key_name) > 0:
        conn.zremrangebyrank(key_name, 0, batch - 1)

Pipelining

Batch commands using pipelines to reduce round-trip time (RTT). This client-side optimization sends multiple commands without waiting for individual responses, receiving replies in bulk. Avoid pipelines exceeding 10,000 commands to prevent memory pressure on client and server.

Transactions

Redis transactions (MULTI/EXEC) provide command grouping but lack rollback capability. Errors during execution don't abort subsequent commands. Use WATCH for optimistic locking in concurrent environments, or Lua scripts for complex atomic operations requiring conditional logic.

Distributed Locks

Implement distributed mutual exclusion using:

SET resource_name unique_value NX PX 30000

Requirements:

• Uniqueness: NX flag ensures exclusive acquisition
• Safety: Auto-expiration prevents deadlock on client failure
• Identity: Unique values (UUIDs) prevent accidental release by other clients
• Atomic Release: Lua script verifies ownership before deletion:

if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end

For multi-master environments, Redlock algorithm requires consensus from N/2+1 independent nodes within TTL constraints, though this adds significant complexity and latency.

Enable lazy freeing for automatic background deletion:

lazyfree-lazy-eviction yes
lazyfree-lazy-expire yes
lazyfree-lazy-server-del yes