When fine-tuning large language models on consumer-grade GPUs with limited VRAM—such as an RTX 4060 Ti with 16GB—memory constraints become a major bottleneck. This article outlines a practical workflow for fine-tuning and deploying the Llama3-Chinese-8B-Instruct model under these conditions.

1. Initial Attempt with LLaMA-Factory

The first approach used LLaMA-Factory, a popular fine-tuning framework. However, training failed due to insufficient GPU memory—approximately 1GB short of the required capacity.

2. Switching to Unsloth for Efficient Fine-Tuning

Unsloth was adopted next, leveraging its memory-optimized training techniques. To stay within VRAM limits, mixed-precision settings were configured as follows:

fp16 = True
bf16 = False

This setup successfully produced a LoRA adapter without exhausting GPU memory.

3. Loading the Model with Transformers

Using Hugging Face transformers, the base model and LoRA weights were loaded as:

from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "path/to/model",
    device_map="auto",
    torch_dtype=torch.float16
)

With device_map="auto", the model loaded successfully but consumed ~13GB of VRAM and exhibited slow inference. Explicitly setting device_map="cuda" triggered an out-of-memory error.

4. Converting to GGUF for Ollama Deployment

Ollama requires models in GGUF format. Direct conversion from the fine-tuned LoRA using:

model.push_to_hub_gguf("hf/model", tokenizer, token="")

failed due to persistent VRAM occupation during the script execution—likely because the model remained loaded after fine-tuning. Adjusting quantization methods (e.g., quantization_method) did not resolve the issue.

5. Manual Conversion via llama.cpp Tools

To bypass the above limitation, the LoRA adapter was first converted to a binary format using a custom script inspired by nous-llama.cpp:

python convert-lora-to-ggml.py --lora_path ./lora_adapter --output ./adapter.bin

6. Quantizing the Base Model

The original Llama3-Chinese-8B-Instruct model was separately converted to GGUF using llama.cpp:

python convert.py /path/to/hf_model
./quantize ./models/ggml-model-f16.gguf ./models/ggml-model-Q4_K_M.gguf Q4_K_M

This followed the official llama.cpp quantization guide.

7. Final Deployement with Ollama

A Modelfile was created to combine the quantized base model and the LoRA adapter:

FROM ./ggml-model-Q4_K_M.gguf
ADAPTER ./adapter.bin
PARAMETER temperature 0.7

After building with ollama create my-model -f Modelfile, the model loaded successfully and ran efficiently on the 16GB GPU.

Open Questions

• Although NVIDIA RTX 40-series GPUs support bfloat16 (bf16), enabling bf16=True during fine-tuning caused instability or errors. The root cause remains unclear—possibly related to library compatibility or CUDA version mismatches.
• Some reports suggest that applying LoRA adapters at inference time in Ollama (without merging) yields suboptimal results. Merging the LoRA into the base model before GGUF conversion may be necessary for full effectiveness—a hypothesis yet to be validated.