Running Qwen2.5 Inference on Ascend 910B NPUs

Configuring the Ascend Environment

On a server equipped with Ascend NPUs, first verify the NPU driver installation:

npu-smi info

Download the appropriate CANN Toolkit (development suite) and Kernel package (CANN operators) for your architecture from the official repository. The following example uses specific version URLs for the Ascend 910B:

wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Milan-ASL/Milan-ASL%20V100R001C19SPC703/Ascend-cann-toolkit_8.0.RC3.alpha003_linux-aarch64.run
wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Milan-ASL/Milan-ASL%20V100R001C19SPC703/Ascend-cann-kernels-910b_8.0.RC3.alpha003_linux-aarch64.run

Proceed to install the Toolkit. Adjust the pip configuration and execute the installer:

sudo sed -i 's/user=true/user=false/' ~/.pip/pip.conf
sudo chmod +x Ascend-cann-toolkit_8.0.RC3.alpha003_linux-aarch64.run && sudo ./Ascend-cann-toolkit_8.0.RC3.alpha003_linux-aarch64.run --install --install-for-all

Install the Kernel package:

sudo chmod +x Ascend-cann-kernels-910b_8.0.RC3.alpha003_linux-aarch64.run && sudo ./Ascend-cann-kernels-910b_8.0.RC3.alpha003_linux-aarch64.run --install --install-for-all

Finally, configure the environment variables by sourcing the setup script:

sudo echo "source /usr/local/Ascend/ascend-toolkit/set_env.sh" >> /etc/profile
source /usr/local/Ascend/ascend-toolkit/set_env.sh

Installing GPUStack

GPUStack is an open-source platform for serving large language models, compatible with various hardware accelerators including Nvidia GPUs, Apple Metal, Huawei Ascend, and Moore Threads. It facilitates running models like Qwen2.5 on Ascend 910B.

Install GPUStack on the Ascend server using the following command. The installation script will prompt for the sudo password.

curl -sfL https://get.gpustack.ai | sh -

For environments with restricted access to GitHub, specify an alternative download source:

curl -sfL https://get.gpustack.ai | sh - --tools-download-base-url "https://gpustack-1303613262.cos.ap-guangzhou.myqcloud.com"

Successful installation is indicated by output similar to:

[INFO]  Install complete.

GPUStack UI is available at http://localhost.
Default username is 'admin'.
To get the default password, run 'cat /var/lib/gpustack/initial_admin_password'.

CLI "gpustack" is available from the command line. (You may need to open a new terminal or re-login for the PATH changes to take effect.)

Retrieve the initial login password:

cat /var/lib/gpustack/initial_admin_password

Access the GPUStack web interface at http://localhost using the username admin and the password obtained above. You will be prompted to set a new password upon first login.

Managing Ascend NPU Resources

GPUStack can manage GPU/NPU resources across multiple Linux, Windows, or macOS devices. To add additional Ascend NPU servers to the cluster, first obtain the cluster token from the primary GPUStack server node:

cat /var/lib/gpustack/token

On each worker node you wish to add, run the installation command with the server URL and token. Replace http://YOUR_IP_ADDRESS with the address of your primary GPUStack server and YOUR_TOKEN with the token from the previous step.

curl -sfL https://get.gpustack.ai | sh - --server-url http://YOUR_IP_ADDRESS --token YOUR_TOKEN --tools-download-base-url "https://gpustack-1303613262.cos.ap-guangzhou.myqcloud.com"

Deploying Qwen2.5 Models

Models can be deployed via the GPUStack web interface. It supports pulling models from HuggingFace, Ollama Library, ModelScope, and private repositories. For networks within China, ModelScope is recommended.

GPUStack offers two inference backends: vLLM and llama-box (an optmiized version of llama.cpp). Current support for Ascend NPUs is provided through the llama-box backend, which requires models in GGUF format.

Due to current limitations in CANN operator support for certain data types, only FP16 precision, Q8_0, and Q4_0 quantized GGUF models are functional on Ascend NPUs. FP16 models are recommended for best results. The following Qwen2.5-Instruct models from ModelScope are compatible:

  • Qwen2.5-0.5B-Instruct-GGUF FP16
  • Qwen2.5-1.5B-Instruct-GGUF FP16
  • Qwen2.5-3B-Instruct-GGUF FP16
  • Qwen2.5-7B-Instruct-GGUF FP16
  • Qwen2.5-14B-Instruct-GGUF FP16
  • Qwen2.5-32B-Instruct-GGUF FP16
  • Qwen2.5-72B-Instruct-GGUF FP16

As an example, the Qwen2.5 72B FP16 model can be deployed and will be scheduled across three Ascend 910B NPUs. The dashboard will show resource allocations such as 140.1 GiB of NPU memory and 8.1 GiB of system RAM.

Performance testing in the Playground shows the Qwen2.5 72B model achieves approximately 6 tokens/second using llama-box on Ascend 910B, with NPU utilization fluctuating between 10% and 30%.

Below is a summary of inference performance for the Qwen2.5 model family on Ascend 910B, including reference data for Q8_0 and Q4_0 quantized versions of the smaller models.

Model Tokens / Second NPU Utilization NPU Memory Usage NPUs Used
Qwen2.5 0.5B FP16 42 tokens/second 6~7% 7% Single Card
Qwen2.5 1.5B FP16 35 tokens/second 11~13% 10% Single Card
Qwen2.5 3B FP16 29 tokens/second 15~16% 15% Single Card
Qwen2.5 7B FP16 32 tokens/second 16~21% 16% Single Card
Qwen2.5 14B FP16 19 tokens/second 19~22% 28% Single Card
Qwen2.5 32B FP16 10.5 tokens/second 10~45% 54% Dual Card
Qwen2.5 72B FP16 6 tokens/second 10~60% 78% Triple Card
Qwen2.5 0.5B Q8_0 6.5 tokens/second 2~5% 6% Single Card
Qwen2.5 0.5B Q4_0 6 tokens/second 4~5% 6% Single Card
Qwen2.5 1.5B Q8_0 3.5 tokens/second 4~11% 8% Single Card
Qwen2.5 1.5B Q4_0 17~18 tokens/second 9~12% 7% Single Card
Qwen2.5 3B Q8_0 3.2 tokens/second 10~15% 10% Single Card
Qwen2.5 3B Q4_0 14.5 tokens/second 8~15% 8% Single Card

Concurrent request testing was performed on the Qwen2.5 0.5B FP16 model, with results as follows:

Concurrent Clients Avg Tokens / Second Total Throughput NPU Utilization NPU Memory Usage
1 39 tokens/second 39 6~7% 7%
2 38 tokens/second 76 6~7% 7%
3 37.66 tokens/second 113 6~7% 7%
4 34.25 tokens/second 137 6~7% 7%
5 31 tokens/second 155 6~7% 7%
6 28.16 tokens/second 169 6~7% 7%
7 27.57 tokens/second 193 6~7% 7%
8 26.87 tokens/second 215 6~7% 7%
9 26 tokens/second 234 6~7% 7%
10 26.9 tokens/second 269 6~7% 7%
20 20.3 tokens/second 406 6~7% 8%
50 10.34 tokens/second 517 3~5% 8%
100 4.17 tokens/second 417 2~5% 9%

The results indicate that hardware performance is not yet fully utilized, suggesting significant room for optimization in CANN operators and inference engine techniques. Future support for Ascend NPUs in GPUStack's vLLM backend is expected to deliver improved inference performance.

GPUStack Feature Overview

  • Heterogeneous GPU Support: Compatible with Nvidia, Apple Metal, Huawei Ascend, and Moore Threads accelerators.
  • Multiple Inference Backends: Supports vLLM and llama-box (llama.cpp) backends, balancing production performance with multi-platform compatibility.
  • Cross-Platform: Runs on Linux, Windows, and macOS, supporting both amd64 and arm64 architectures.
  • Diverse Model Types: Accommodates LLMs, VLMs, Embedding models, and Reranker models.
  • Multiple Model Repositories: Pull models from HuggingFace, Ollama Library, ModelScope, and private repositories.
  • Advanced Scheduling: Offers compact, spread, label-based, and GPU-specific scheduling policies.
  • Ditsributed Inference: Automatically splits large models across multiple GPUs on different hosts.
  • CPU Inference: Supports running models on CPU resources via mixed GPU/CPU or pure CPU modes.
  • Model Comparison: Playground provides a side-by-side view to compare responses and performance metrics across different models, weights, prompts, quantizations, hardware, and backends.
  • Comprehensive Monitoring: Delivers detailed performance, utilization, status, and usage metrics for both GPU/NPU devices and LLM serving.

As an open-source project, GPUStack offers a straightforward setup to deploy a private, enterprise-ready model serving platform.

Tags: Ascend 910B Qwen2.5 LLM Inference GPUStack CANN

Posted on Sun, 25 Oct 2026 16:13:33 +0000 by TF@TheMoon

Hot Tags

©2026 Freaks City