A production alert came in at midnight while the on-call engineer was asleep. After connecting to the environment, it became clear that one microservice was experiencing a steady increase in file descriptor usage, eventually leading to an OutOfMemoryError and service crash. This microservice establishes SSH connections during operation, and since similar incidents had occurred before, the initial hypothesis was that some SSH connection was not being properly closed.

Guessing wouldn't yield results, so the first step was to examine the logs for error messages. After thorough investigation, a peculiar error appeared in the logs: NoClassDefFoundError. The initial understanding of this error was superficial—its first interpretation was that a class file was missing. Navigating to the microservice directory and checking the class file showed it existed. Then, checking file permissions with ls -la confirmed they were correct. At this point, the situation seemed puzzling.

With no clear answer, the only path forward was to dive into the source code. Analyzing the stack trace pointed to specific source locations, and additional research into the meaning of NoClassDefFoundError eventually revealed the root cause.

Understanding NoClassDefFoundError

The key to understanding NoClassDefFoundError lies in the following explanation: this error indicates that the JVM previously attempted to load a class but failed for some reason. Now that the class needs to be used again, the loading is tirggered once more. However, because the previous loading attempt failed, the JVM does not retry the loading process—it simply throws the NoClassDefFoundError immediately.

Consider this example demonstrating the behavior:

public class Demo {
    public static void main(String[] args) {
        try {
            Calculator calc1 = new Calculator();
        } catch (Throwable t) {
            System.out.println("First attempt failed: " + t);
        }
        Calculator calc2 = new Calculator();
    }
}

class Calculator {
    static int value = 10 / 0;
}

Running this code produces an exception during the first instantiation of Calculator because the static initializer attempts to evaluate 10 / 0, which throws an ArithmeticException. When the second instantiation occurs, the JVM remembers the previous failure and immediately throws NoClassDefFoundError without attempting to load the class again.

The Actual Production Issue

Production code rarely exhibits such obvious bugs. The actual problematic code structure looked something like this:

public class ConnectionManager {
    private static final DataSource datasource = DatabaseContext.getDataSource();
}

public class DatabaseContext {
    public static <T> T getDataSource() {
        return applicationContext.getBean(DataSource.class);
    }
}

public class RequestHandler {
    public static void processRequest() {
        ConnectionManager.execute();
    }
}

public class SshSession implements Closeable {
    private SshConnection connection;
    
    public void execute() {
        try {
            connection = sshFactory.createConnection();
        } finally {
            ConnectionManager.closeConnection(connection);
        }
    }
}

The ConnectionManager utility class depends on DatabaseContext to retrieve a Spring-managed DataSource bean. During service startup, RequestHandler.processRequest() was invoked, which triggered ConnectionManager class loading and consequently the initialization of the datasource field. At this point, the Spring application context had not finished initializing, so DatabaseContext.getDataSource() threw an exception, causing the first initialization attempt to fail.

After the service started successfully, other components calling ConnectionManager methods would encounter NoClassDefFoundError because the JVM refused to retry class initialization. This rendered all ConnectionManager methods unusable. When SSH sessions completed normally, they attempted to call ConnectionManager.closeConnection() to clean up resources. Since this method was unavailable, the SSH connections could not be properly closed, leading to the continuous growth in file descriptor count and ultimately the service crash.