As AI tools like ChatGPT become ubiquitous, concerns regarding user data privacy have intensified. There is a growing demand for solutions that leverage generative AI capabilities without transmitting sensitive information to external servers. PrivateGPT addresses this by enabling users to interact with large language models (LLMs) entirely offline, processing local documents without compromising data sovereignty.

Overview of PrivateGPT

PrivateGPT is a production-ready AI solution designed to operate without an internet connection. It allows users to query local documents using LLM power while ensuring 100% privacy—no data leaves the local execution environment.

The system functions as a service that encapsulates AI RAG (Retrieval-Augmented Generation) primitives within a comprehensive API framework. Its highly modular, supporting various local and remote LLM providers, embedding models, and vector stores, which can be swapped without modifying the core codebase.

System Architecture

The backbone of the project consists of FastAPI for the server layer and LlamaIndex for data orchestration. The architecture exposes a set of APIs divided into two logical tiers:

High-Level API: This layer abstracts the complexity of the RAG pipeline:

• Document Ingestion: Handles parsing, chunking, metadata extractino, embedding generation, and storage automatically.
• Contextual Chat & Completion: Manages context retrieval, prompt engineering, and response generation based on ingested documents.

Low-Level API: Designed for advanced users requiring custom pipelines:

• Embedding Generation: Creates vector representations for specific text inputs.
• Chunk Retrieval: Returns the most relevant text segments from the document store based on a query.

Additionally, the project includes a Gradio-based web interface for testing and utility scripts for tasks like batch model downloads and directory monitoring.

Setup and Installation

Before proceeding, ensure the environment meets the following requirements:

Python >= 3.11
make
poetry

Follow these steps for a standard local deployment:

# Clone the repository
git clone https://github.com/imartinez/privateGPT
cd privateGPT

# Setup Python version
pyenv install 3.11
pyenv local 3.11

# Install project dependencies
poetry install --with ui,local

# Download required Embedding and LLM models
poetry run python scripts/setup

# Optional: Enable GPU support for Mac Metal (or adjust for other platforms)
export CMAKE_ARGS="-DLLAMA_METAL=on"
pip install --force-reinstall --no-cache-dir llama-cpp-python

# Launch the server
PGPT_PROFILES=local make run

# Verify GPU usage in logs (e.g., ggml_metal_add_buffer)
# Access the interface at http://localhost:8001/

For optimal performance, GPU acceleration is recommended. The system is compatible with llama.cpp models. Common choices include:

1. TheBloke/Llama-2-7B-chat-GGUF
2. TheBloke/Mistral-7B-Instruct-v0.1-GGUF

Operational Modes

Once the server is running at http://localhost:8001/, you can select from three operational modes via the UI:

• Document Q&A: Answers questions by retrieving context from the ingested files. It maintains conversation history for contextual awareness.
• Semantic Search: Performs a quick lookup to return the top 4 relevant text chunks along with source document details.
• Direct LLM Chat: Engages in a standard conversation with the language model without accessing the document database, relying only on the chat history.