Overview

What is Caching?

A cache is defined as storage for content that may be needed again in the future, allowing for rapid retrieval. Caches are temporary collections of data that are either duplicates of information located elsewhere or results of computations. Data already present in a cache can be accessed repeatedly at minimal time and resource cost.

Why Implement Caching?

• Reduces redundant computation
• Improves service throughput
• Reduces load on underlying data systems

Common Caching Approaches

• Distributed caching (Memcached, Redis, Tair)
• Local caching (in-memory only/in-memory with disk persistence)

Key Cache Terminology

• Cache Hit: When a requested key returns a valid data entry from the cache. The most important metric for cache efficiency is the hit rate.
• Hot Data: Most applications follow an 80/20 rule where a small portion of data is accessed frequently. This data is referred to as hot data.
• Eviction: Since cache capacity is limited, when new elements are added and the cache is full, older elements must be removed. Eviction strategies significantly impact hit rates, with common approaches including FIFO (First-In-First-Out), LRU (Least Recently Used), and LFU (Least Frequently Used).

Guava Cache

Guava Cache is designed similarly to ConcurrentHashMap, offering comprehensive functionality with simplicity of use. A common use case is as a ConcurrentHashMap alternative when you need a cache with eviction capabilities.

Key Features

• Supports LRU eviction strategy but doesn't allow custom strategies
• Eviction based on element insertion time or access time
• Configurable maximum size
• Removal event listeners
• LoadingCache for defining element loading logic

Basic Implementation

// Create a cache with maximum size of 1000 items
// Items expire 60 seconds after write
DataStore<String, Object> dataStore = DataStoreBuilder.<String, Object>newBuilder()
        .maximumCapacity(1000)
        .expireAfterDuration(60, TimeUnit.SECONDS)
        .build();

// Store a value
dataStore.put("user:123", new UserProfile());

// Retrieve a value
UserProfile profile = (UserProfile) dataStore.get("user:123");

Caffeine

Caffeine can be considered an evolution of Guava Cache, implementing high-performance read/write interfaces and providing the W-TinyLFU eviction algorithm for superior cache hit rates. Starting with Spring 5, Caffeine replaced Guava as the default cache implementation.

Usage Interface

Caffeine's interface is largely compatible with Guava's implementation approach.

// Initialize a cache with capacity and expiration
MemoryStore<String, Object> memoryStore = MemoryStore.<String, Object>newInstance()
        .maxItems(10_000)
        .expireAfterWrite(60, TimeUnit.SECONDS)
        .initialize();

// Add an entry
memoryStore.store("session:abc", new SessionData());

// Retrieve an entry
SessionData session = (SessionData) memoryStore.fetch("session:abc");

Performance Characteristics

Caffeine's performance is optimized through several key design choices: sequential access queues, asynchronous read/write operations, tiered time wheeels, and code generation techniques. According to official benchmarks, Caffeine significantly outperforms both Guava Cache and even ConcurrentHashMap without caching capabilities.

Performance Benchmarks

• Read-heavy workload (100% reads): With 8 threads performing concurrent reads on a capacity-limited cache.
• Mixed workload (75% reads, 25% writes): With 6 threads reading and 2 threads writing concurrently to a capacity-limited cache.
• Write-heavy workload (100% writes): With 8 threads performing concurrent writes to a capacity-limited cache.

Hit Rate Optimization

Caffeine uses the Window TinyLFU (W-TinyLFU) strategy due to its high hit rates and lower memory overhead. W-TinyLFU is an evolution of LFU that addresses the issue where historically frequent data might occupy cache space, preventing new hot data from being stored.

CountMin Sketch for Reduced Counter Space

CountMin Sketch uses a two-dimensional array (implemented as a single long array) and multiple hash functions to approximately record the access frequency of each key, similar to the Bloom filter concept. When a key is accessed, it uses predefined hash functions to find positions in the array and update values. The minimum value among these positions is considered the key's access frequency. Multiple hash functions and taking the minimum value help reduce errors caused by hash conflicts.

Freshness Mechanism

Since counters in CountMin Sketch only increase, old data might persist while new hot data gets evicted before its frequency increases. TinyLFU resets all values in the Sketch when the total recorded frequencies exceed 10 times the cache's maximum size. Resetting involves dividing all recorded frequencies by 2.

TinyLFU Algorithm

TinyLFU serves as a pre-filter for other cache eviction policies (like LRU). When new data enters the cache, it uses the frequency data from CountMin Sketch to compete with data being evicted from the cache, with the winner being stored.

W-TinyLFU Enhancement

W-TinyLFU adds a Window Cache in front of TinyLFU to address the retention of sudden hot data. The Window Cache uses LRU and starts at only 1% of the total cache size. Data evicted from Window Cache enters TinyLFU for competition with data evicted from MainCache. The winner returns to MainCache while the loser is permanently evicted.

MainCache is divided into Probation and Protected segments, similar to JVM's young and old generations. New data first enters the Probation segment, and when accessed again, moves to the Protected segment.

Ehcache

Ehcache is an open-source, standards-based, robust, reliable, fast, simple, and lightweight Java distributed cache. It integrates with other frameworks and serves as Hibernate's default CacheProvider. Ehcache implements the JSR107 specification (JCache) and differs from Guava and Caffeine with its support for multiple storage media: heap, off-heap, disk, and distributed configurations with multi-tier combinations.

Basic Implementation

// Initialize CacheManager
CacheManager cacheManager = CacheManagerFactory.<String, Object>newInstance()
        .withPersistence(new File(getStoragePath()))
        .initialize();

// Configure resource pools
StorageConfig storageConfig = StorageConfigBuilder.newInstance()
        .heapEntries(5000)
        .offHeapSize(40, MemoryUnit.MEGABYTES)
        .diskSize(10, MemoryUnit.GIGABYTES)
        .create();

// Configure cache with expiration
CacheConfig<String, Object> cacheConfig = CacheConfigBuilder.<String, Object>newInstance(
        String.class, Object.class, storageConfig)
        .withTimeToLiveExpiration(Duration.ofHours(1))
        .create();

// Create cache instance
DataCache<String, Object> dataCache = cacheManager.createCache(
        "primaryCache", cacheConfig);

Multi-tier Read/Write Operations

In a multi-tier architecture, during a put operation, data is first written to the authoritative tier (disk) and values in other tiers are invalidated. During a get operation, the system attempts retrieval from top to bottom until reaching the authoritative tier. After retrieving data from a lower tier, it's promoted to upper tiers. Given that each put operation accesses disk, an efficient serializer is crucial for performance.

Comparison Summary

Feature Comparison of Three Java Caching Solutions