When building a Flink project with Spring Boot 3.4.6 and JDK 17, I encountered persistent class loading problems after packaging. The application would fail at runtime with ClassNotFoundException errors, even though all required JAR files were present in the final artifact. This issue stemmed from Service Provider Interface (SPI) definitions not being properly merged during the build process.

The Core Problem

The initial build configuration used spring-boot-maven-plugin to create an executable JAR. While this approach works for typical Spring Boot microservices, it falls short for Flink applications due to how it handles META-INF/services/ resources. Multiple dependencies contain service provider configuration files with the same name, and the Spring Boot plugin's packaging strategy doesn't merge these files—it simply overwrites them. This breaks Java's ServiceLoader mechanism, which relies on aggregating all provider implementations across the classpath.

The ServiceLoader API, introduced in Java 6 and enhanced in modular Java, discovers and loads implementations of service interfaces at runtime. When these META-INF/services/ entries are lost, Flink's internal factories and connectors cennot be instantiated, leading to runtime failures.

Implementing the Shade Plugin Solution

The maven-shade-plugin provides a robust solution through its ServicesResourceTransformer, which intelligently merges service descriptor files from all included dependencies. Here's the complete build configuration:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.13.0</version>
            <configuration>
                <source>${java.version}</source>
                <target>${java.version}</target>
                <encoding>UTF-8</encoding>
                <parameters>true</parameters>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.6.0</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <artifactSet>
                            <excludes>
                                <exclude>com.google.code.findbugs:jsr305</exclude>
                            </excludes>
                        </artifactSet>
                        <filters>
                            <filter>
                                <artifact>*:*</artifact>
                                <excludes>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <transformers>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>com.dataflow.pipeline.StreamProcessor</mainClass>
                            </transformer>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
    <resources>
        <resource>
            <directory>src/main/resources</directory>
            <includes>
                <include>**/*</include>
            </includes>
        </resource>
        <resource>
            <directory>external-libs</directory>
            <targetPath>/BOOT-INF/lib/</targetPath>
            <includes>
                <include>**/*.jar</include>
            </includes>
        </resource>
    </resources>
    <finalName>pipeline-${project.version}</finalName>
</build>

The critical element is the ServicesResourceTransformer, which automatically concatenates all META-INF/services/ files with identical names, ensuring every service provider is registered.

Plugin Comparison: Shade vs Spring Boot

For Flink-based data processing jobs, maven-shade-plugin offers distinct advantages:

1. Superior SPI Handling

While Spring Boot's plugin can potentially be configured with custom layouts to address SPI merging, it requires additional complexity. The Shade plugin handles this transparently out-of-the-box.

2. Faster Startup Times

Shaded JARs exhibit quicker class loading because dependencies are unpacked at the root level rather than nested inside BOOT-INF/lib/. This flat structure eliminates the overhead of Spring Boot's custom classloader hierarchy, which is unnecessary for Flink deployments where the job JAR is submitted to an existing cluster.

3. Leaner Artifact Structure

Unlike Spring Boot's executable archives, shaded JARs contain only classes and resources without the launcher infrastructure, making them more suitable for distributed processing frameworks.

When Spring Boot Plugin Excels

The Spring Boot Maven plugin shines in scenarios requiring its ecosystem features. For instance, the build-info goal generates a build-info.properties file for actuator endpoints:

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <executions>
        <execution>
            <goals>
                <goal>build-info</goal>
            </goals>
            <configuration>
                <additionalProperties>
                    <encoding.source>UTF-8</encoding.source>
                    <java.version>${java.version}</java.version>
                </additionalProperties>
            </configuration>
        </execution>
    </executions>
</plugin>

This capability remains valuable for monitoring and observability in Spring Boot applications, though less critical for pure Flink jobs.

Compiler Plugin Configuration

Both approaches benefit from explicit maven-compiler-plugin settings to control compilation behavior:

• Preserving parameter names with -parameters for reflection-based frameworks
• Enabling lint checks via -Xlint for code quality
• Targeting specific Java versions and character encodings
• Supporting JPMS module system configurations