Managing heterogeneous accelerators for Retrieval-Augmented Generation pipelines requires a unified serving layer. GPUStack provides an open-source infrastructure to aggregate NVIDIA, Apple Silicon, Ascend, and Moore Threads hardware into a single cluster, enabling on-premise deployment of the three foundational components for RAG: conversational LLMs, text embedding models, and cross-encoder rerankers.

Environment Provisioning

Deployment begins with initializing the control plane. On Unix-like systems, execute the installer script. If direct access to upstream repositories is restricted, route binary fetches through a regional mirror by exporting the base URL beforehand:

export MIRROR_ENDPOINT="https://gpustack-1303613262.cos.ap-guangzhou.myqcloud.com"
curl -fsSL https://get.gpustack.ai | sh -s -- --tools-download-base-url "${MIRROR_ENDPOINT}"

For Windows anvironments, launch an elevated PowerShell session and run:

$MirrorUrl = "https://gpustack-1303613262.cos.ap-guangzhou.myqcloud.com"
$Installer = (Invoke-WebRequest -Uri "https://get.gpustack.ai" -UseBasicParsing).Content
Invoke-Expression "& { $Installer } --tools-download-base-url '$MirrorUrl'"

Upon succesfull initialization, the service binds to port 80. Retrieve the generated administrator credentials:

# Linux/macOS
sudo cat /var/lib/gpustack/initial_admin_password

# Windows
Get-Content "$env:APPDATA\gpustack\initial_admin_password" -Raw

Access the web dashboard via http://localhost, authenticate as admin, and immediately rotate the default passphrase.

Cluster Expansion and Worker Registration

To distribute inference workloads across additional machines, generate a join token from the primary node:

cat /var/lib/gpustack/token

Apply this token on target worker nodes to attach them to the cluster. Replace the placeholders with your actual network configuration:

curl -fsSL https://get.gpustack.ai | sh -s -- \
  --server-url http://${MASTER_NODE_HOST} \
  --token ${CLUSTER_AUTH_TOKEN} \
  --tools-download-base-url "${MIRROR_ENDPOINT}"

Windows workers follow an identical parameter structure via PowerShell. Once registered, the dashboard will reflect the aggregated compute capacity.

Serving the RAG Model Trio

Navigate to the Models interface to provision the required architectures. The platform automatically routes GGUF artifacts to the llama-box runtime (optimized for cross-platform CPU/GPU execution) and standard safetensors/PyTorch weights to vLLM (Linux-only, high-throughput). For domestic network constraints, pulling from ModelScope is recommended.

Deploy the following GGUF-quantized variants to cover the complete RAG stack:

• Qwen/Qwen2.5-7B-Instruct-GGUF (Generative component)
• gpustack/bge-m3-GGUF (Vectorization component)
• gpustack/bge-reranker-v2-m3-GGUF (Semantic ranking component)

Ensure the GGUF format toggle is active during provisioning. For vision-language requirements, select non-GGUF checkpoints to trigger the vLLM backend automatically. All deployed services expose OpenAI and Jina-compatible REST endpoints.

Orchestrating with Dify

Containerize the application layer using Docker Compose. Isolate the stack from the inference cluster to prevent port collisions:

git clone --depth 1 --branch 0.11.1 https://github.com/langgenius/dify.git
pushd dify/docker
cp .env.example .env
# Adjust HOST_PORT in .env if port 80 is occupied
docker compose up -d --remove-orphans
popd

Initialize the Dify administrator account at http://localhost. Navigate to Settings > Model Providers and select the GPUStack connector. Configure three distinct entries:

1. LLM Connector: Input the deployment identifier (e.g., qwen2.5-7b-instruct), the cluster endpoint (http://<CLUSTER_IP>), and a valid API key. Set context window to 8192 and maximum generation tokens to 2048.
2. Embedding Connector: Register the vector model (e.g., bge-m3) using the same endpoint and credentials. Align the context limit to 8192.
3. Rerank Connector: Add the cross-encoder (e.g., bge-reranker-v2-m3) with matching network parameters and a context ceiling of 8192.

Verify connectivity by refreshing the provider dashboard. All three endpoints should display active status.

Constructing the Retrieval Pipeline

Within the Dify Knowledge interface, instantiate a new dataset and upload target documents. During the indexing configuraton, assign the previously registered embedding model. Enable hybrid search and toggle the reranking step, linking it to the cross-encoder provider.

Once the chunking and vectorization jobs complete, execute a retrieval test. The pipeline will perform initial dense/sparse lookup, followed by the reranker to score and reorder candidates based on semantic relevance.

Finalize the workflow by creating a Chat application. Attach the prepared knowledge base to the context window. During inference, the system will vectorize user queries, retrieve and rerank document chunks, and pass the curated context to the generative LLM for grounded responses. Any external framework supporting OpenAI or Jina API specifications can consume these endpoints identically.

Platform Capabilities Overview

• Hardware Aggregation: Unified scheduling across NVIDIA, Apple Metal, Ascend, and Moore Threads accelerators.
• Dual Runtime Architecture: Automatic routing between vLLM for high-concurrency Linux deployments and llama-box for cross-OS compatibility and CPU fallback.
• Cross-OS Compatibility: Native binaries for Linux, Windows, and macOS across amd64 and arm64 architectures.
• Model Diversity: Native support for text generation, vision-language, dense embeddings, and cross-encoder rerankers.
• Registry Flexibility: Direct pulling from Hugging Face, Ollama, ModelScope, or private OCI/S3-compatible repositories.
• Advanced Scheduling: Policies include bin-packing, spread distribution, node-label affinity, and explicit GPU pinning.
• Tensor Parallelism: Automated model sharding across multiple hosts when single-device VRAM is insufficient.
• CPU Offloading: Hybrid GPU/CPU execution or pure CPU inference for resource-constrained environments.
• Evaluation Workspace: Side-by-side playground for benchmarking latency, throughput, and output quality across different quantizations, prompts, and backends.
• Telemetry: Granular metrics for VRAM allocation, compute utilization, request latency, and token throughput.