Preparing the GPU layer with GPUStack

Start a GPUStack server container to manage GPU resources and model deployments across multiple nodes.

docker run -d --name gpustack-server \
  --restart unless-stopped \
  -p 80:80 \
  -v gpustack-data:/var/lib/gpustack \
  -v /data/gpustack_cache:/var/lib/gpustack/cache \
  gpustack/gpustack:v2.1.0 \
  --bootstrap-password "admin123" \
  --debug

Visit http://<server-ip> and sign in with admin / admin123.

Defining a cluster and connecting workers

GPUStack organises resources into clusters. Upon first login, follow the prompt to create your initial cluster. After cluster creation, add a worker node by executing the provided command on a machine with NVIDIA drivers and Docker.

docker run -d --name gpustack-worker \
   -e "GPUSTACK_RUNTIME_DEPLOY_MIRRORED_NAME=gpustack-worker" \
   -e "GPUSTACK_TOKEN=<worker-token>" \
   --restart=unless-stopped \
   --privileged \
   --network=host \
   --volume /var/run/docker.sock:/var/run/docker.sock \
   --volume gpustack-worker-data:/var/lib/gpustack \
   --volume /data/gpustack_cache:/var/lib/gpustack/cache \
   --runtime nvidia \
   gpustack/gpustack:v2.1.0 \
   --server-url http://<server-ip> \
   --worker-ip <worker-ip>

Deploying inference models

Head to Deployments and use Deploy Now to browse the model catalogue. The following models are used in this setup:

• Qwen3-Reranker-4B
• Qwen3-Embedding-4B
• Qwen3.5-35B-A3B

For the Qwen3.5-35B-A3B deployment, configure the environment variable PYPI_PACKAGES_INSTALL to include transformers upgrades if necessary.

After each deployment finishes, interact with the model through the Playground to confirm its operational.

Retrieving model endpoint details

Open the Routes page, click the context menu beside a route, and choose API Access Info. Note the Base URL, model name, and an API key.

Base URL: http://192.168.50.14/v1
Model Names:
  qwen3.5-35b-a3b
  qwen3-reranker-4b
  qwen3-embedding-4b
API Key: gpustack_...

Generate an API key from the UI if one does not exist.

Setting up MaxKB

Launch MaxKB with a single Docker command:

docker run -d --name=maxkb --restart=always \
  -p 8080:8080 -v ~/.maxkb:/opt/maxkb 1panel/maxkb

Default credentials are admin / MaxKB@123..; change the password on first login.

Registering GPUStack models in MaxKB

Navigate to the Model section and click Add Model. Provide the model name and base model; the API URL and API Key fields appear after you type the base model and press Enter.

• Fill API URL with http://<gpustack-host>/v1
• Provide the API Key collected earlier

Add entries for both qwen3-embedding-4b and qwen3-reranker-4b. For the reranker model, enable Generic Proxy because MaxKB invokes /v2/rerank.

Creating an intelligent knowledge assistant

Ingesting documentation

Go to Knowledge, create a new knowledge base, and select Web Knowledge. Input a documentation URL; MaxKB will fetch and parse the content automatical.

Building an Agent

Open the Agent page and create a new agent. Choose the LLM (qwen3.5-35b-a3b), the embedding model (qwen3-embedding-4b), and the reranker (qwen3-reranker-4b). Attach the previously created knowledge base. Adjust the system prompt and parameters as needed, then publish the agent.

Once published, start a conversation to verify that the assistant retrieves accurate information from the ingested documents.