Addressing Heap Pressure with Off-Heap Storage

Large in-memory caches frequently trigger prolonged Garbage Collection (GC) pauses in Java applications. When cached objects dominate the heap, tuning JVM parameters often yields diminishing returns. Relocating cache storage to off-heap memory isolates it from the GC lifecycle, leveraging physical RAM directly without impacting heap management. The Off-Heap Cache (OHC) library provides a production-ready implementation for this architecture.

Off-heap memory operates outside the JVM's menaged heap boundareis. Its capacity is constrained only by the host machine's physical RAM rather than -Xmx settings. This approach is particularly effective when local cache volume causes frequent GC cycles, heap exhaustion, or latency spikes, and when distributed caching introduces unacceptable network overhead.

Implementing OHC with Custom Serialization

Because off-heap memory stores raw bytes, OHC requires explicit serialization and deserialization strategies. The library does not impose a default format, allowing developers to optimize for their specific data structures. Below is a complete implementation demonstrating cache initialization and a custom UTF-8 string serializer.

<dependency>
    <groupId>org.caffinitas.ohc</groupId>
    <artifactId>ohc-core</artifactId>
    <version>0.7.4</version>
</dependency>

import org.caffinitas.ohc.CacheSerializer;
import org.caffinitas.ohc.OHCache;
import org.caffinitas.ohc.OHCacheBuilder;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;

public class OffHeapCacheExample {

    private static final CacheSerializer<String> UTF8_SERIALIZER = new CacheSerializer<String>() {
        @Override
        public void serialize(String value, ByteBuffer buffer) {
            byte[] payload = value.getBytes(StandardCharsets.UTF_8);
            buffer.putInt(payload.length);
            buffer.put(payload);
        }

        @Override
        public String deserialize(ByteBuffer buffer) {
            int length = buffer.getInt();
            byte[] payload = new byte[length];
            buffer.get(payload);
            return new String(payload, StandardCharsets.UTF_8);
        }

        @Override
        public int serializedSize(String value) {
            return value.getBytes(StandardCharsets.UTF_8).length + Integer.BYTES;
        }
    };

    public static void main(String[] args) {
        OHCache<String, String> cache = OHCacheBuilder.<String, String>newBuilder()
                .keySerializer(UTF8_SERIALIZER)
                .valueSerializer(UTF8_SERIALIZER)
                .capacity(100L * 1024 * 1024) // 100 MB off-heap limit
                .build();

        cache.put("app_config", "production_mode");
        System.out.println("Cache hit: " + cache.get("app_config"));
    }
}

The API mirrors standard java.util.Map operations, but all data transitions through the provided serializer before residing in native memory. Custom objects can be supported by implementing equivalent byte-conversion logic using libraries like Kryo, Protobuf, or manual ByteBuffer manipulation.

On-Heap vs. Off-Heap Memory Behavior

Comparing standard heap-based caching against OHC highlights the architectural differences in memory consumption and failure modes. The following test continuously inserts 1 MB payloads into a ConcurrentHashMap with a restricted heap.

// JVM Args: -Xms128m -Xmx128m
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public class HeapPressureTest {
    private static final Map<String, byte[]> heapStore = new ConcurrentHashMap<>();

    public static void main(String[] args) throws InterruptedException {
        Thread.sleep(5000); // Allow profiler attachment
        int index = 0;
        while (true) {
            heapStore.put("entry_" + index, new byte[1024 * 1024]);
            index++;
        }
    }
}

Under a 128 MB heap limit, this process rapidly exhausts available space, triggering repeated GC cycles before terminating with an OutOfMemoryError. Memory profiling shows a sharp spike followed by immediate collapse.

Transitioning the same workload to OHC alters the behavior significantly. The JVM heap remains stable while the operating system's native memory absorbs the allocation.

// JVM Args: -Xms128m -Xmx128m
import org.caffinitas.ohc.OHCache;
import org.caffinitas.ohc.OHCacheBuilder;

public class OffHeapPressureTest {
    public static void main(String[] args) throws InterruptedException {
        Thread.sleep(5000);
        
        OHCache<String, byte[]> nativeStore = OHCacheBuilder.<String, byte[]>newBuilder()
                .keySerializer(UTF8_SERIALIZER)
                .valueSerializer(RAW_BYTE_SERIALIZER) // Assumes equivalent implementation
                .capacity(2L * 1024 * 1024 * 1024) // 2 GB off-heap reservation
                .build();

        int index = 0;
        while (true) {
            nativeStore.put("entry_" + index, new byte[1024 * 1024]);
            index++;
        }
    }
}

With off-heap storage, the JVM heap footprint remains flat. Memory consumption scales linearly against the host's available RAM until the configured capacity threshold is reached. OHC then applies its eviction policies rather than crashing the JVM. Monitoring requires observing OS-level memory metrics instead of standard JMX heap indicators.

Internal Allocation Strategies and Native Memory Management

OHC provides two primary allocation implementations: linked and chunked. The linked strategy allocates discrete native memory blocks per cache entry, optimizing for medium to large payloads. The chunked strategy pre-allocates memory segments to reduce fragmentation for small entries, but remains experimental. Production deployments should default to the linked implementation for stability.

A critical design decision in OHC is the deliberate avoidance of java.nio.ByteBuffer.allocateDirect(). Standard direct byte buffers tie native memory cleanup to the Java GC via Cleaner or PhantomReference mechanisms. This introduces several operational risks:

• Unpredictable Reclamation: Native memory is only freed when the associated Java wrapper is garbage collected, creating a lag between cache eviction and actual OS memory release.
• Full GC Triggers: When direct memory limits (-XX:MaxDirectMemorySize) are approached, the JVM may forcefully invoke System.gc(), causing stop-the-world pauses.
• Global Synchronization: The JDK tracks direct allocations using a globally synchronized list, which can become a contention point under high allocation throughput.

To bypass these limitations, OHC interacts directly with native allocators using sun.misc.Unsafe or JNA (com.sun.jna.Native). This grants deterministic control over malloc and free equivalents, ensuring immediate memory reclamation upon cache eviction without GC interference.

import sun.misc.Unsafe;
import java.lang.reflect.Field;

public class NativeAccessor {
    public static Unsafe resolveUnsafe() {
        try {
            Field unsafeField = Unsafe.class.getDeclaredField("theUnsafe");
            unsafeField.setAccessible(true);
            return (Unsafe) unsafeField.get(null);
        } catch (Exception ex) {
            throw new IllegalStateException("Unable to initialize Unsafe API", ex);
        }
    }
}

By invoking Unsafe.allocateMemory() and Unsafe.freeMemory() directly, OHC manages the complete lifecycle of cached data. For Linux environments, the library documentation recommends preloading jemalloc to replace the default glibc malloc. jemalloc significantly reduces memory fragmentation and improves allocation throughput under concurrent workloads, aligning with OHC's low-latency design goals.