In the Ascend hardware ecosystem, enference engines like vLLM have demonstrated excellent performance for generation tasks and are widely adopted. However, for retrieval-related scenarios such as Embedding and Reranker tasks, specialized inference components like mis-tei offer efficient capabilities that are better suited to these specific use cases.

mis-tei focuses on text vector generation and reordering capabilities, with broad applications in retrieval-augmented generation (RAG), semantic search, and similar domains.

During practical implementation, we identified several challenges:

• The official image was functional but presented high integration complexity
• Path dependencies, parameter configurations, and operational methods introduced certain barriers
• Additional adaptation was required when integrating with GPUStack

To address these issues, we performed repackaging and simplifciation of the official image and made it available as a community image, significantly reducing integration complexity.

This article demonstrates how to integrate the mis-tei backend into GPUStack using the community image for high-performance Embedding/Reranker model inference.

Image Overview

1. Ascend Official Images

swr.cn-south-1.myhuaweicloud.com/ascendhub/mis-tei:7.3.0-300I-Duo-aarch64
swr.cn-south-1.myhuaweicloud.com/ascendhub/mis-tei:7.3.0-800I-A2-aarch64
swr.cn-south-1.myhuaweicloud.com/ascendhub/mis-tei:7.3.0-800I-A3-aarch64

2. GPUStack Community Images (Recommended)

swr.cn-south-1.myhuaweicloud.com/gpustackcommunity/mis-tei:7.3.0-300I-Duo-aarch64
swr.cn-south-1.myhuaweicloud.com/gpustackcommunity/mis-tei:7.3.0-800I-A2-aarch64
swr.cn-south-1.myhuaweicloud.com/gpustackcommunity/mis-tei:7.3.0-800I-A3-aarch64

3. Optimization Details

Compared to the official images, the community version includes the following improvements:

• ✅ Streamlined startup scripts with optimized default parameter configurations
• ✅ Support for arbitrary parameter passthrough to enhance flexibility
• ✅ Ready-to-use integration with GPUStack, lowering the entry barrier

For specific implementation details of the repackaging process, refer to the community backend repository: https://github.com/gpustack/community-inference-backends/tree/main/mis-tei

Integrating the mis-tei Backend

In GPUStack, you can entegrate mis-tei through the custom backend functionality.

Implementation Steps

1. Navigate to the Inference Backend page
2. Click Add Backend → Custom in the upper right corner
3. Fill in parameters as shown in the following example

⚠️ Note: Select the appropriate image based on your Ascend device model

YAML Configuration (Direct Import)

backend_name: mis-tei-custom
health_check_path: /health
default_run_command: --model-id {{model_path}} -p {{port}}
default_env:
  ENABLE_BOOST: "True"
  AUTO_TRUNCATE: "true"
version_configs:
  7.3.0-a2:
    image_name: swr.cn-south-1.myhuaweicloud.com/gpustackcommunity/mis-tei:7.3.0-800I-A2-aarch64
    custom_framework: cann
  7.3.0-a3:
    image_name: swr.cn-south-1.myhuaweicloud.com/gpustackcommunity/mis-tei:7.3.0-800I-A3-aarch64
    custom_framework: cann
  7.3.0-310p:
    image_name: swr.cn-south-1.myhuaweicloud.com/gpustackcommunity/mis-tei:7.3.0-300I-Duo-aarch64
    custom_framework: cann
default_version: 7.3.0-a2

⚠️ Note: When copying YAML from public platforms, invisible characters (like NBSP) may be included, causing parsing failures. It's recommended to copy from the original GitHub file: https://github.com/gpustack/community-inference-backends/blob/main/mis-tei/spec.yaml

Parameter Explanation

• image_name: Uses the community repackaged image
• custom_framework: Specifies the Ascend CANN runtime environment
• default_run_command: Unified startup parameter template
• health_check_path: Health check endpoint

After configuration, the backend should appear as follows:

Model Deployment

The following demonstrates deployment using the BGE-M3 embedding model as an example:

1. Open the model library and filter for Embedding models
2. Switch the inference engine to mis-tei
3. Click Deploy and wait for the model status to change to Running
4. Open the Testbed to perform validation

Model Support Information

For model types supported by mis-tei and related parameter configurations, refer to the official documentation:

👉 https://www.hiascend.com/developer/ascendhub/detail/07a016975cc341f3a5ae131f2b52399d

Conclusion

Through repackaging and adaptation of the official Ascend mis-tei image, we have implemented a more lightweight and user-friendly integration solution in GPUStack:

• 🚀 Significantly reduces deployment and configuration complexity
• 📦 Provides ready-to-use community images
• 🔧 Simplifies model service integration workflow
• ⚡ Optimized for RAG scenarios, better connecting retrieval and generation pipelines

In the next version, this backend will be integrated as a built-in community backend in GPUStack, enabling users to activate it with a single click.

Additional Resources

For more community backend projects, visit:

👉 https://github.com/gpustack/community-inference-backends

Contributions are welcome to help improve the GPUStack ecosystem.