IKAnalyzer is a lightweight Chinese text segmentation library built on Java. It serves as a Chinese language analysis component that combines dictionary-based tokenization with grammatical analysis algorithms, originally developed as an extension to the Lucene search engine framework. The 2012 release introduced an improved algorithm for handling tokenization ambiguities, making it particularly useful for applications requiring precise Chinese text processing.

Several compelling reasons make IKAnalyzer an excellent choice for Chinese text segmentation in modern applications:

• The library implements a unique "forward iteration finest-grained segmentation algorithm" that supports both precise and intelligent tokenization modes, allowing developers to choose the appropriate approach based on their specific requirements.
• Performance testing on standard hardware (Core2 i7 3.4GHz dual-core, 4GB RAM, Windows 7 64-bit, Sun JDK 1.6_29 64-bit) demonstrates processing speeds of approximately 1.6 million Chinese characters per second (3000KB/s), making it suitable for high-throughput applications.
• The intelligent tokenization mode handles basic ambiguity resolution and combines numeric expressions appropriately during output.
• A multi-processor analysis architecture provides robust support for mixed-content scenarios including English letters, numeric values, and Chinese vocabulary within the same text stream.
• Optimized dictionary storage mechanisms minimize memory footprint, enabling efficient operation even in resource-constrained environments.

Include the following Maven dependencies in your pom.xml file to integrate IKAnalyzer into your Spring Boot project. Note that we explicitly exclude Lucene dependencies to avoid version conflicts with those managed by your project's dependency management:

<dependency>
    <groupId>com.janeluo</groupId>
    <artifactId>ikanalyzer</artifactId>
    <version>2012_u6</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.lucene</groupId>
            <artifactId>lucene-core</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.lucene</groupId>
            <artifactId>lucene-queryparser</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.lucene</groupId>
            <artifactId>lucene-analyzers-common</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>org.apache.lucene</groupId>
    <artifactId>lucene-queryparser</artifactId>
    <version>7.3.0</version>
</dependency>

Dictionary Configuration

IKAnalyzer utilizes two essential dictionary types that control how text gets segmented:

Extended Dictionary

The extended dictionary ensures that multi-word phrases are treated as single tokens rather than being broken into smaller pieces. This proves invaluable when you need to preserve proper nouns, brand names, or domain-specific terminology in their complete form. For instance, if "Chinese Taiwan" exists in your extended dictionary, the text "Chinese Taiwan in East China Sea" will be tokenized as "Chinese Taiwan | in | East China Sea" instead of the granular "Chinese | Taiwan | in | East | China | Sea".

Stop Word Dictionary

The stop word dictionary allows you to specify common words that should be filtered out during tokenization. These typically include functional words, particles, and other high-frequency terms that carry little semantic meaning for search or analysis purposes. When a stop word is encountered, it is completely removed from the output rather than being included as a token.

Configuration Files Setup

Create the following configuration files in your Spring Boot project's src/main/resources directory:

IKAnalyzer Configuration XML

Create a file named IKAnalyzer.cfg.xml that points to your custom dictionaries:

<?xml version="1.0" encoding="UTF-8"?>

<properties>
    <comment>IK Analyzer Configuration - Custom Dictionaries</comment>
    <entry key="ext_dict">/custom.dic</entry>
    <entry key="ext_stopwords">/filter.dic</entry>
</properties>

Extended Dictionary File (custom.dic)

Define phrases that should remain unsplit in this file:

spring framework
cloud native
distributed system

Stop Word Dictionary File (filter.dic)

Specify words to be removed from tokenization results:

the
and
of
in
on

Implementation

The following utility class provides a clean interface for performing Chinese text segmentation in your Spring Boot application:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.wltea.analyzer.core.IKSegmenter;
import org.wltea.analyzer.core.Lexeme;

import java.io.StringReader;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

public final class ChineseSegmenter {

    private static final Logger LOG = LoggerFactory.getLogger(ChineseSegmenter.class);
    
    private ChineseSegmenter() {
        // Utility class - prevent instantiation
    }
    
    /**
     * Segments input text into individual Chinese tokens using IKAnalyzer.
     *
     * @param inputText the Chinese text to be segmented
     * @return a list of tokens extracted from the input text
     */
    public static List<String> tokenize(String inputText) {
        if (inputText == null || inputText.trim().isEmpty()) {
            return Collections.emptyList();
        }
        
        List<String> tokens = new ArrayList<>();
        StringReader reader = new StringReader(inputText.trim());
        
        // Initialize IKSegmenter with smart mode disabled for maximum precision
        IKSegmenter segmenter = new IKSegmenter(reader, false);
        
        Lexeme nextLexeme;
        while ((nextLexeme = segmenter.next()) != null) {
            tokens.add(nextLexeme.getLexemeText());
        }
        
        return tokens;
    }
}

Testing the Implementation

Execute the following test code to verify that the segmentation operates correctly:

public static void main(String[] args) {
    String sampleText = "Welcome to Spring Boot's official documentation website";
    
    List<String> tokens = ChineseSegmenter.tokenize(sampleText);
    System.out.println("Segmentation Results:");
    tokens.forEach(System.out::println);
}

Sample Output:

Segmentation Results:
welcome
to
spring
boot
official
documentation
website