What is Spring AI?

Spring AI addresses a fundamental challenge in the AI landscape: the lack of a standardized interface for interacting with various Large Language Models (LLMs). Each LLM provider (OpenAI, DeepSeek, Zhipu, etc.) has its own unique API format and conventions. Spring AI abstracts this complexity by introducing a unified ChatModel interface. This allows developers to write code against a single, consistent API, regardless of the underlying model. Switching between different LLMs becomes a matter of configuration rather than code changes.

Additionally, Spring AI simplifies prompt engineering through template mechanisms, allowing prompts to be stored externally and populated with variables. It also provides built-in integrations for vector databases like PGVector and Milvus, and encapsulates Function Calling, enabling LLMs to invoke external tools via annotations.

How to Integrate with DeepSeek using Spring AI?

Spring AI offers two primary abstractions for interacting with LLMs: ChatModel and ChatClient.

• ChatModel: This is the core, low-level interface that directly encapsulates the communication logic with an LLM. It provides basic methods like call(Prompt) and stream(Prompt). When using ChatModel, the developer is responsible for constructing the prompt and parsing the raw response.
• ChatClient: This is a higher-level, fluent API built on top of ChatModel. It offers a builder-style interface that automates complex tasks such as prompt handling, response parsing, parameter configuration, conversation memory, and Retrieval-Augmented Generation (RAG). Think of ChatModel as a manual transmission and ChatClient as an automatic transmission.

It's worth noting a common configuration issue in Spring AI 1.1, particularly with the OpenAI module. When customizing the base-url, a bug could cause malformed URLs, such as double slashes (//) or repeated version paths (/v1/v1), leading to 404 errors or connection failures.

How is the "Unified Abstraction" Achieved for Multiple Models?

The "unified abstraction" is implemented using the Adapter Pattern. Spring AI defines a single ChatModel interface. Each LLM provider (e.g., OpenAI, DeepSeek) has its own adapter class (e.g., OpenAiChatModel, DeepSeekChatModel) that implements this interface. These adapters translate the provider's specific API calls into the standard ChatModel methods.

Developers configure the desired model in the application.yml file and inject the appropriate service. To dynamically switch between models, a strategy pattern can be employed. Here is an example of a factory method that selects a ChatModel based on a configuration key:

// A factory method to select the appropriate ChatModel based on configuration
public ChatModel selectChatModel(String modelProvider) {
    Map<String, ChatModel> modelMap = Map.of(
        "openai", openAIModel,
        "deepseek", deepSeekModel,
        "zhipu", zhipuModel
    );
    ChatModel selectedModel = modelMap.get(modelProvider.toLowerCase());
    if (selectedModel == null) {
        throw new IllegalArgumentException("Unsupported model provider: " + modelProvider);
    }
    return selectedModel;
}

How are Streaming Responses Implemented in Spring AI?

Streaming allows an LLM to send back a response token by token, rather than waiting for the entire responce to be generated. The ChatClient's .stream() method returns a Flux<ChatResponse>, a reactive stream from the Project Reactor library.

The flow is as follows:

1. Your application calls chatModel.stream(prompt).
2. Spring AI sends a request to the LLM with stream=true.
3. The LLM begins sending individual tokens.
4. The response is formatted using Server-Sent Events (SSE).
5. Spring AI packages each token into a ChatResponse object.
6. These ChatResponse objects are pushed downstream through the Flux pipeline.
7. Your application subscribes to the Flux to receive each chunk of text as it arrives.

Flux is the reactive container that manages this continuous stream of data.

Common Flux Operations

// Obtain a stream of responses
Flux<String> responseStream = chatClient.prompt()
    .user(userInput)
    .stream()
    .content();

// Collect all chunks into a single list (this is a blocking operation)
List<String> allChunks = responseStream.collectList().block();

// Concatenate all chunks into a single string
String fullResponse = responseStream.reduce("", (accumulator, chunk) -> accumulator + chunk).block();

// Apply a timeout to the stream
responseStream.timeout(Duration.ofMinutes(1));

// Provide a fallback value in case of an error
responseStream.onErrorReturn("An error occurred during generation.");

// Process each received chunk
responseStream.doOnNext(chunk -> log.info("Received chunk: {}", chunk));

Distinction from SseEmitter: While Flux<ChatResponse> is the stream of data provided by Spring AI, it cannot be directly returned to a web client. An SseEmitter (from Spring MVC) is the tool used to push these chunks to the frontend over an HTTP connection.

What is the Advisor Mechanism in Spring AI?

An Advisor in Spring AI acts as an interceptor or enhancer for ChatClient calls, similar to a Web Filter or an AOP around advice. It allows for pluggable, composable logic to be executed before and after an LLM call. This is useful for tasks like modifying prompts, injecting context, managing memory, or performing RAG. Spring AI provides separate interfaces for synchronous and streaming calls: CallAdvisor and StreamAdvisor.

Advisors are chained together to form a processing pipeline.

// A chain of advisors to process a prompt
chatClient.prompt()
    .user("Explain quantum computing.")
    .advisors(contextInjector, promptValidator, memoryManager)
    .call()
    .content();

How is Function Calling Implemented?

Function Calling enables an LLM to request the execution of an external tool (e.g., a weather API, a database query) when it encounters a question it cannot answer from its training data. The process is:

1. The user's question is sent to the LLM.
2. The LLM determines a tool is needed and returns a special function_call instruction.
3. Spring AI automatically invokes the corresponding Java method.
4. The method's result is sent back to the LLM.
5. The LLM uses this new information to generate the final answer.

Spring AI leverages Spring's @Bean and @Description annotations to define these callable functions.

// Define a tool for the AI to use
@Bean
@Description("Retrieves user profile information from the database based on a user ID.")
public Function<UserProfileRequest, UserProfile> fetchUserProfile() {
    return request -> {
        // Business logic to fetch and return user data
        return userProfileService.findById(request.getUserId());
    };
}

// A simple data class for the request
public record UserProfileRequest(String userId) {}

// A simple data class for the response
public record UserProfile(String name, String email) {}

Why is the buildChatModel Factory Pattern Used?

The design of creating ChatModel instances on-demand with a factory promotes session isolation and dynamic configuration. This approach is crucial for several reasons:

• Session Isolation: Each request gets its own instance, preventing context from leaking between different users or API calls.
• Dynamic Configuration: It allows for easy switching of models, API keys, and parameters (like temperature or max tokens) on a per-request basis.
• Resource Management: Instances are created and destroyed for each request, preventing resource leaks and memory bloat in high-concurrency scenarios. A singleton ChatModel could lead to thread-safety issues and incorrect state management.

What is LangChain4j?

LangChain4j is the Java port of LangChain, one of the most popular AI frameworks in the Python ecosystem. It brings the core concepts of LangChain—such as Chains, Agents, Memory, Tools, and RAG—into the Java world. Developers familiar with Python's LangChain will find LangChain4j's concepts and APIs very intuitive.

How to Handle Token Limit Exceedances?

When the input prompt exceeds a model's token limit, several strategies can be employed:

• Sliding Window: Maintain a fixed-size window of the most recent messages in the conversation history.
• Summarization: If a single message is too long, it can be summarized by the LLM before being included in the prompt.
• Pre-estimation: Estimate the token count of the prompt before sending it to the model and truncate or adjust it if necessary.