Understanding Redis Cache Penetration

Cache penetration occurs when a high volume of requests target data that is not present in the cache. When a cache lookup fails (a cache miss), the request typically proceeds to query the underlying database.

The risk emerges when numerous concurrent requests simultaneously seek non-existent data. Since these keys are absent from Redis, each request bypasses the cache and directly burdens the database. This pattern can overwhelm database resources, similar to a cache avalanche but triggered by missing data rather than expired keys.

Common Causes of Concurrent Requests for Non-Existent Data

A foundational principle for engineers is to assume clients may act with malicious intent. Requests should not be presumed to be benign. Scenarios include competitive attacks, automated scraping, or deliberate Distributed Denial-of-Service (DDoS) attempts aimed at system exhaustion.

Illustrative Scenario

Consider an API endpoint for fetching user details:

https://api.example.com/users/12345

A legitimate request for an existing user ID would first check the cache. Malicious traffic, however, might target patently invalid IDs concurrently:

https://api.example.com/users/-100 https://api.example.com/users/99999999999999999999

These IDs—negative or exceeding database constraints—will not be found in the database. Since they are also absent from the cache, every such request forces a direct database query, consuming resources.

Mitigation Techniques for Cache Penetration

While blocking source IPs at the infrastructure level can stop simple attacks, its ineffective against distributed attacks from numerous compromised hosts. Application-level defenses are required.

1. Request Validation and Filtering

Implement input validation at the application's edge. Reject requests with parameters that violate business rules (e.g., negative IDs, non-numeric values, or excessively long strings) before performing any cache or database operation.

2. Caching Non-Existent Keys (Cache Nulls)

This is the primary defense. When a database query confirms a requested record does not exist, store a placeholder value for that key in Redis. Subsequent requests for the same invalid key will hit this cached placeholder, preventing repeated database access.

• The cached value can be a special marker (e.g., "NULL", an empty JSON object {}).
• Assign a relatively short Time-To-Live (TTL), such as 5-10 minutes, to these placeholder keys. This limits memory usage and allows recovery if the data later becomes valid.

Example Implementation

async function fetchUserData(userId) {
  const cacheKey = `user:${userId}`;
  const placeholder = "##NULL##";

  // 1. Attempt to retrieve from cache
  let cachedData = await redisClient.get(cacheKey);

  if (cachedData !== null) {
    // If cached value is our placeholder, return a "not found" response
    if (cachedData === placeholder) {
      return { status: 404, message: "User not found" };
    }
    // Otherwise, return the valid cached data
    return JSON.parse(cachedData);
  }

  // 2. Input validation (e.g., reject invalid IDs)
  if (!isValidUserId(userId)) {
    // Cache the invalid key with a short TTL
    await redisClient.setex(cacheKey, 300, placeholder);
    return { status: 400, message: "Invalid user ID" };
  }

  // 3. Query the database
  const userRecord = await database.query('SELECT * FROM users WHERE id = ?', [userId]);

  // 4. Handle database result
  if (userRecord.length === 0) {
    // Record does not exist: cache the placeholder
    await redisClient.setex(cacheKey, 300, placeholder);
    return { status: 404, message: "User not found" };
  }

  // 5. Record exists: cache the actual data
  await redisClient.setex(cacheKey, 3600, JSON.stringify(userRecord[0]));
  return userRecord[0];
}

function isValidUserId(id) {
  // Example validation: must be a positive integer within a sane range
  const numId = Number(id);
  return Number.isInteger(numId) && numId > 0 && numId < 1000000;
}