Exception Manifestation

2018-03-16 18:53:59,501 ERROR [DefaultMessageListenerContainer-1] (com.bill99.asap.service.CryptoClient.seal(CryptoClient.java:34))- null
java.lang.NullPointerException
    at java.net.InetAddress$Cache.put(InetAddress.java:779) ~[?:1.7.0_79]
    at java.net.InetAddress.cacheAddresses(InetAddress.java:858) ~[?:1.7.0_79]
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1334) ~[?:1.7.0_79]
    at java.net.InetAddress.getAllByName0(InetAddress.java:1248) ~[?:1.7.0_79]
    at java.net.InetAddress.getAllByName(InetAddress.java:1164) ~[?:1.7.0_79]
    at java.net.InetAddress.getAllByName(InetAddress.java:1098) ~[?:1.7.0_79]
    at java.net.InetAddress.getByName(InetAddress.java:1048) ~[?:1.7.0_79]
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:220) ~[?:1.7.0_79]
    at sun.net.NetworkClient.doConnect(NetworkClient.java:180) ~[?:1.7.0_79]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) ~[?:1.7.0_79]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) ~[?:1.7.0_79]
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) ~[?:1.7.0_79]
    at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[?:1.7.0_79]
    at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[?:1.7.0_79]
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:997) ~[?:1.7.0_79]
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933) ~[?:1.7.0_79]
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:851) ~[?:1.7.0_79]
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1092) ~[?:1.7.0_79]
    at org.springframework.ws.transport.http.HttpUrlConnection.getRequestOutputStream(HttpUrlConnection.java:81) ~[spring-ws-core.jar:1.5.6]
    at org.springframework.ws.transport.AbstractSenderConnection$RequestTransportOutputStream.createOutputStream(AbstractSenderConnection.java:101) ~[spring-ws-core.jar:1.5.6]
    at org.springframework.ws.transport.TransportOutputStream.getOutputStream(TransportOutputStream.java:41) ~[spring-ws-core.jar:1.5.6]
    at org.springframework.ws.transport.TransportOutputStream.write(TransportOutputStream.java:60) ~[spring-ws-core.jar:1.5.6]

Once this exception occurs, it cascades continuously, eventually rendering system nodes unavailable.

Root Cause Analysis

Examining the JDK Source Code

The NullPointerException originates from InetAddress$Cache.put at line 779. Investigating the implementation reveals the problematic section:

if (policy != InetAddressCachePolicy.FOREVER) {
    LinkedList<String> expiredKeys = new LinkedList<>();
    long currentTime = System.currentTimeMillis();
    for (String key : cache.keySet()) {
        CacheEntry entry = cache.get(key);

        if (entry.expiration >= 0 && entry.expiration < currentTime) {
            expiredKeys.add(key);
        } else {
            break;
        }
    }

    for (String key : expiredKeys) {
        cache.remove(key);
    }
}

The null pointer exception occurs when accessing entry.expiration, indicating that cache.get(key) returns null. However, the cache insertion logic always creates a new CacheEntry object:

CacheEntry entry = new CacheEntry(addresses, expiration);
cache.put(host, entry);

This observation suggests a race condition. The keySet() iteration occurs simultaneously with a remove operation on the underlying LinkedHashMap, causing get(key) to return null for keys that have been removed by other threads.

Cache Access Points

The InetAddress class provides two synchronized methods for cache manipulation:

private static void ensureCacheInitialized() {
    assert Thread.holdsLock(addressCache);
    if (addressCacheInitialized) {
        return;
    }
    unknownAddresses = new InetAddress[1];
    unknownAddresses[0] = implementation.anyLocalAddress();

    addressCache.put(implementation.anyLocalAddress().getHostName(),
                     unknownAddresses);

    addressCacheInitialized = true;
}

private static void storeInCache(String hostname,
                                  InetAddress[] addresses,
                                  boolean successful) {
    hostname = hostname.toLowerCase();
    synchronized (addressCache) {
        ensureCacheInitialized();
        if (successful) {
            addressCache.put(hostname, addresses);
        } else {
            negativeCache.put(hostname, addresses);
        }
    }
}

private static InetAddress[] retrieveFromCache(String hostname) {
    hostname = hostname.toLowerCase();

    synchronized (addressCache) {
        ensureCacheInitialized();

        CacheEntry entry = addressCache.get(hostname);
        if (entry == null) {
            entry = negativeCache.get(hostname);
        }

        if (entry != null) {
            return entry.addresses;
        }
    }

    return null;
}

Both storeInCache and retrieveFromCache are synchronized on addressCache, preventing concurrent access issues under normal circumstances.

Discovery of Reflection-Based Cache Access

Upon further investigation, business code was discovered accessing the internal cache through reflection:

static {
    Class<?> inetAddressClass = java.net.InetAddress.class;
    Field cacheField = inetAddressClass.getDeclaredField("addressCache");
    cacheField.setAccessible(true);
    Object cacheContainer = cacheField.get(inetAddressClass);

    Class<?> cacheContainerClass = cacheContainer.getClass();
    Field underlyingMapField = cacheContainerClass.getDeclaredField("cache");
    underlyingMapField.setAccessible(true);
    Map<String, Object> cacheMap = (Map) underlyingMapField.get(cacheContainer);
}

public void clearCacheEntry(String host) {
    cacheMap.remove(host);
}

This reflection-based access bypasses the synchronization mechanisms built into the JDK, directly manipulating the LinkedHashMap without any locking. When one thread removes entries while another thread iterates using keySet(), the race condition manifests as a NullPointerException.

Why Continuous Exceptions Occur

The continuous exception pattern occurs because LinkedHashMap maintains internal state that becomes inconsistent under concurrent modifications. Consider this test case:

public class ConcurrentMapTest {

    public static void main(String[] args) throws InterruptedException {
        final LinkedHashMap<Integer, DataObject> dataMap = new LinkedHashMap<>();
        
        // Single thread performing put operations
        new Thread(() -> {
            for (int i = 0; i < 2000; i++) {
                dataMap.put(new Random().nextInt(1000), 
                           new DataObject(new Random(100).nextInt()));
            }
        }).start();
        
        // Multiple threads performing remove operations
        for (int i = 0; i < 100; i++) {
            new Thread(() -> {
                for (int j = 0; j < 500; j++) {
                    dataMap.remove(new Random().nextInt(1000));
                }
            }).start();
        }
        
        Thread.sleep(2000);
        System.out.println("mapSize=" + dataMap.keySet().size() + ", keys=" + dataMap.keySet());
        
        for (Integer key : dataMap.keySet()) {
            System.out.println(dataMap.get(key));
        }
    }
}

class DataObject {
    private int value;

    public DataObject(int value) {
        this.value = value;
    }

    public int getValue() {
        return value;
    }
}

Running this test produces output demonstrating the race condition:

mapSize=0,[121, 517, 208]
null
null
null

The LinkedHashMap.size() method returns 0 eventhough keySet() contains entries, and get() consistently returns null. This behavior occurs because LinkedHashMap is not thread-safe, and concurrent modifications corrupt its internal data structures.

Resolution

Approach 1: Synchronized Access

Apply the same synchronization pattern used by the JDK when accessing the cache through reflection:

private static final Object cacheLock = new Object();

public void clearCacheEntry(String host) {
    synchronized (cacheLock) {
        cacheMap.remove(host);
    }
}

Approach 2: Avoid Direct Cache Manipulation

Leverage established libraries for DNS cache manipulation instead of implementing custom solutions. The Alibaba DNS Cache Mainpulator project provides thread-safe operations:

DnsCacheManipulator.putDnsCache("example.com", new InetAddress[] { 
    InetAddress.getByName("127.0.0.1") 
});

DnsCacheManipulator.removeDnsCache("example.com");

Reference: https://github.com/alibaba/java-dns-cache-manipulator

Key Takeaways

1. Internal JDK data structures often have synchronization mechanisms that reflection-based access bypasses entirely
2. LinkedHashMap and similar collections are inherently thread-unsafe and require external synchronization for concurrent access
3. Race conditions in cache implementations can produce intermittent failures that quickly escalate to system-wide unavailability
4. When troubleshooting mysterious null pointer exceptions in standard library code, investigate whether application code is improperly accessing internal state through reflection

Technical Notes

The root cause stems from a combination of factors: the application used reflection to access InetAddress.addressCache, the underlying cache storage uses a non-thread-safe LinkedHashMap, and cache modifications occurred without proper synchronization. This resulted in internal map corruption where keys existed but returned null values, causing persistent NullPointerException exceptions that cascaded across all DNS resolution attempts.

The fix requires either implementing proper synchronization around all cache access patterns or, preferably, delegating DNS cache management to thread-safe libraries that handle these edge cases correctly.