GPUStack is an open-source platform designed to simplify the deployment and management of large language models (LLMs) across heterogeneous GPU clusters. While public cloud LLM APIs are widely accessible, organizations seeking private, on-premises LLM hosting face significant complexity in infrastructure setup, model orchestration, and resource optimization. GPUStack addresses these challenges by abstracting away the underlying infrastructure stack—eliminating the need for Kubernetes expertise or manual AI toolchain integration.

The system unifies GPUs from diverse vendors—including NVIDIA, AMD, Intel, Apple Silicon, and others—across macOS, Windows, and Linux hosts into a single managed pool. Administrators can deploy GGUF-formatted models directly from Hugging Face or Ollama Library, while developers interact with these models through a fully OpenAI-compatible REST API.

Core Capabilities

Heterogeneous GPU Clustering
GPUStack aggregates compute resources from mixed environments—desktops, workstations, and servers—regardless of operating system or GPU brand. Initial releases support Apple Macs and NVIDIA-equipped Windows/Linux machines, with broader vendor support planned.

Inteligent Model Deployment
Models are automatically scheduled to nodes with sufficient resources. GPUStack leverages LLaMA.cpp as its default inference backend, supporting GGUF models. When GPU memory is constrained, it seamlessly falls back to hybrid CPU+GPU or CPU-only inference, enabling deployment even in edge or resource-limited settings.

OpenAI-Compatible API
Applications integrate with private LLMs using standard OpenAI API calls. A built-in playground allows developers to test prompts, adjust parameters, and generate ready-to-use client code in Python, Node.js, or cURL.

Observability and Metrics
Administrators monitor GPU utilization, temperature, and memory usage in real time. Developers track token throughput, request rates, and consumption metrics—enabling performance tuning and future auto-scaling capabilities.

Role-Based Access Control
GPUStack enforces authentication and RBAC: administrators manage models and infrastructure, while developers consume APIs. Applications authenticate via API keys tied to user permissions.

Use Cases

• Cost-effective LLMaaS: Pool idle developer workstations (Mac/Windows/Linux) into a shared inference cluster.
• Edge LLM Inference: Deliver LLM capabilities in disconnected or low-resource environments using CPU fallback.
• Secure On-Prem LLM Hosting: Run sensitive workloads without relying on external cloud providers.

Getting Started

Install Server & First Worker (Linux/macOS)

curl -sfL https://get.gpustack.ai | sh -

Access the dashboard at http://<server-ip>. Retrieve the initial admin password:

cat /var/lib/gpustack/initial_admin_password

Add Additional Workers On new nodes:

curl -sfL https://get.gpustack.ai | sh - --server-url http://<server-ip> --token <worker-token>

Obtain <worker-token> from the server:

cat /var/lib/gpustack/token

Windows Installation (Admin PowerShell)

Invoke-Expression (Invoke-WebRequest -Uri "https://get.gpustack.ai" -UseBasicParsing).Content

Retrieve credentials:

Get-Content -Path "$env:APPDATA/gpustack/initial_admin_password" -Raw

Deploy a Model

1. Log in as admin and navigate to Models.
2. Select a GGUF model from Hugging Face or Ollama Library.
3. GPUStack auto-schedules the model to an appropriate worker.

Integrate via API Generate an API key under user settings. Example cURL request:

export GPUSTACK_API_KEY=your_api_key
curl http://<server-ip>/v1-openai/chat/completions \
  -H "Authorization: Bearer $GPUSTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

For detailed installation guides and architecture diagrams, visit docs.gpustack.ai. Source code and issue tracking are available at github.com/gpustack/gpustack.