A colleague from another team reported that during project releases or restarts (using kill -15), C-end business services often experienced issues leading to financial loss.

After hearing about potential money loss, I immediately took responsibility, gathered details, and started investigating.

Problem Analysis

Based on the descriptoin and my knowledge of the business, I quickly identified that Kafka message loss was causing the C-end problems.

Current Consumption Architecture

From the diagram, several factors can cause message loss in this scenario:

1. Kafka commits offsets once per second
2. The queue still holds unprocessed tasks
3. The worker thread pool hasn’t finished tasks pulled from the queue (pulls one task at a time)

The core issue: Because the C-end business treats non-real-time messages (with in minutes) as meaningless, automatic offset commits actually fit the requirements. The real problem during deployment is that the single-threaded consumer keeps polling messages and writing them into the queue, while the thread pool hasn’t finished processing the queued tasks.

Consumption Architecture Refactoring

1. Redesign the consumption flow
2. Add a JVM shutdown hook to set an isRunning flag to false on shutdown, stopping the single consumer thread from polling Kafka messages
3. Gracefully shut down the worker thread pool

// Example comparing shutdown() vs shutdownNow()
ThreadPoolExecutor pool =
   new ThreadPoolExecutor(1, 1, 30, TimeUnit.SECONDS, new LinkedBlockingQueue<>());
AtomicInteger counter = new AtomicInteger();
for (int i = 0; i < 100; i++) {
   pool.execute(() -> {
       try {
           System.out.println(LocalDateTime.now() + " => " + counter.incrementAndGet());
           Thread.sleep(1000L);
       } catch (Exception e) {
           e.printStackTrace();
       }
   });
}

Thread.sleep(5000L);
pool.shutdown();
// pool.shutdownNow();
System.out.println("Shutdown triggered on thread pool");

A new problem arises: if the JVM shutdown hook calls shutdown() on the worker pool, and Spring beans are already destroyed, the remaining tasks that depend on those beans will fail (the destruction order can vary; details are easy to find).

This reminded me of @PostConstruct’s twin annotation: @PreDestroy, introduced by JSR-250. Spring supports it for lifecycle callbacks. Here's a test snippet:

ThreadPoolExecutor executor =
        new ThreadPoolExecutor(1, 1, 30, TimeUnit.SECONDS, new LinkedBlockingQueue<>());

@PostConstruct
public void onStart() {
    AtomicInteger counter = new AtomicInteger();
    for (int i = 0; i < 100; i++) {
        executor.execute(() -> {
            try {
                System.out.println(Instant.now() + " ===> " + counter.incrementAndGet());
                Thread.sleep(1000L);
            } catch (Exception e) {
                e.printStackTrace();
            }
        });
    }
}

@PreDestroy
public void onDestroy() {
    executor.shutdown();
}

// Test endpoint to trigger JVM exit
@GetMapping("/exit")
public void exitApp() {
    System.exit(0);
}

The test still failed: logs showed the process dying while tasks were being processed. (That didn’t match the earlier shutdown() test where tasks completed before the JVM exited.)

After some thought, I realized this behavior makes sense—a pool with too many tasks shouldn’t prevent the process from terminating indefinitely. But how to handle the remaining work? Fortunately, Doug Lea had already considered this:

// Improved shutdown logic
@PreDestroy
public void onDestroy() {
   executor.shutdown();
   try {
       if (executor.awaitTermination(5, TimeUnit.SECONDS)) {
           System.out.println("All tasks finished normally");
       } else {
           System.out.println("Timed out waiting for tasks");
       }
   } catch (InterruptedException e) {
       System.out.println("Interrupted while waiting for executor");
       Thread.currentThread().interrupt();
       executor.shutdownNow();
   }
}

This approach looks cleaner: after shutdown(), we wait up to N seconds (returns immediately if no tasks remain). You can tune this timeout based on business characteristics.

Yet adding this boilerplate to every important thread pool is tedious. How does Spring handle graceful pool shutdown? By implementing DisposableBean via ThreadPoolTaskExecutor. Its parent class ExecutorConfigurationSupport checks the waitForTasksToCompleteOnShutdown flag during destruction. When true, it calls shutdown() and then uses the awaitTerminationSeconds property to call awaitTermination on the underlying ExecutorService.

Now we can refactor the worker thread pool by configuring these two properties and letting Spring handle the shutdown routine.

Conclusion

Using Spring’s thread pool executor and setting these two parameters enables graceful shutdown:

1. waitForTasksToCompleteOnShutdown – triggers shutdown() during bean destruction
2. awaitTerminationSeconds – waits for a defined period after shutdown() to allow remaining tasks to complete

ExecutorService.awaitTermination is useful but shouldn’t be overused. If many pools are configured with a long wait and still contain a large number of tasks at shutdown, the kill -15 time can increase, giving the impression that the process cannot be terminated.