The llama.cpp project provides a high-performance C++ implemnetation for running Large Language Models (LLMs) like Llama 2 with minimal overhead. It is designed for efficient inference on various hardware setups, ranging from standard consumer laptops to cloud environments, without requiring heavy dependencies.

Building llama.cpp from Source

To get started, you need to clone the repository and compile the binaries for your specific operating system.

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

Compilation on macOS

For macOS users, utilizing Apple Silicon's Metal API significantly accelerates inference.

LLAMA_METAL=1 make

Compilation on Windows

On Windows, you should use CMake to generate the build files. Ensure you have a C++ compiler and CMake installed.

cmake -B build
cmake --build build --config Release

The compiled executables will be located in build/bin/Release.

Acquiring and Setting Up Models

Llama 2 models are available in several formats. The most common for local execution is the GGUF format, which is optimized for llama.cpp.

Using Pre-quantized Models

You can download pre-quantized GGUF files directly from repositories like Hugging Face (e.g., TheBloke's Llama-2-7B-Chat-GGUF). Place the .gguf file into the models/ directory of your llama.cpp folder.

Converting Original Meta Weights

If you have the original model weights from Meta, you must convert and quantize them. First, install the Python requirements:

pip3 install -r requirements.txt

Convert the model to a high-precision format first:

python3 convert.py ./models/llama-2-13b-chat/ --outfile ./models/llama-2-13b-chat/fp16-model.bin --outtype f16

Then, quantize the model to 4-bit to reduce memory usage:

./quantize ./models/llama-2-13b-chat/fp16-model.bin ./models/llama-2-13b-chat/q4_0-model.bin q4_0

Running Inference via Command Line

Once you have your model file ready, you can start an interactive session. Use the -ngl flag to offload layers to your GPU (Metal on Mac or CUDA on Windows).

./llama-cli -m ./models/llama-2-13b-chat/q4_0-model.bin -n 512 --repeat_penalty 1.0 --color -i -ngl 32

Programmatic Access via Python

The llama-cpp-python library allows you to integrate Llama 2 directly into Python applications.

from llama_cpp import Llama

# Initialize the model instance
llm_engine = Llama(
    model_path="./models/llama-2-7b.Q4_0.gguf",
    n_ctx=2048,
    n_threads=4
)

# Execute inference
query = "Explain the concept of recursion in programming."
response_data = llm_engine(
    query,
    max_tokens=150,
    stop=["User:", "\n"],
    echo=False
)

print(response_data["choices"][0]["text"])

Deploying a Local Web Server

llama.cpp includes a built-in HTTP server that exposes an API compatible with common LLM interfaces.

./llama-server -m ./models/llama-2-7b.Q4_0.gguf --port 9000 --host 0.0.0.0

You can then interact with the server using curl or a Python script:

import requests

def get_llm_response(user_input):
    api_url = "http://localhost:9000/completion"
    payload = {
        "prompt": user_input,
        "n_predict": 256
    }
    
    request_headers = {"Content-Type": "application/json"}
    response = requests.post(api_url, json=payload, headers=request_headers)
    return response.json().get("content")

if __name__ == "__main__":
    print(get_llm_response("What are the benefits of C++?"))

Inference using Hugging Face Transformers

For systems with significant VRAM, you can use the standard transformers library to run the Hugging Face version of Llama 2.

from transformers import AutoTokenizer, LlamaForCausalLM
import torch

model_id = "meta-llama/Llama-2-7b-chat-hf"

def run_hf_inference(text_prompt):
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    model = LlamaForCausalLM.from_pretrained(
        model_id, 
        torch_dtype=torch.float16, 
        device_map="auto"
    )

    inputs = tokenizer(text_prompt, return_tensors="pt").to("cuda")
    output_tokens = model.generate(**inputs, max_new_tokens=100)
    
    return tokenizer.decode(output_tokens[0], skip_special_tokens=True)

print(run_hf_inference("How does a transformer model work?"))