Model Preparation

There are several reliable methods to download pre-trained models:

1. ModelScope (recommended for fast download speeds and includes many restricted models)
2. Hugging Face mirror sites (requires model access permisions)
3. Public cloud storage resources (use tools like XShell for faster uploads)

Dataset Preparation

Two primary dataset formats are supported. Refer to the sample datasets in the repository for structure. Important considerations:

• Test the combined length of instruction and input fields to determine appropriate max_seq_length
• Add custom dataset metadata in data_info.json. Example format:

"CustomDataset": {
    "file_name": "training_data.json",
    "columns": {
        "prompt": "instruction",
        "query": "input",
        "response": "output",
        "history": "history"
    }
}

Environment Setup

Follow these steps to configure the training environment:

# Using CUDA 12.1
conda create -n llama_factory python=3.11
cd LLaMA-Factory
conda install pytorch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -e ".[torch,metrics]"
pip install wandb
DS_BUILD_CPU_ADAM=1 pip install deepspeed==0.14.0

Distributed Training Configuration

For single-node multi-GPU training:

• Remove evaluation-related parameters when working with large datasets
• Set the correct number of GPUs in LLaMA-Factory/examples/accelerate/fsdp_config.yaml:

num_processes: 4 # total number of available GPUs

15. Adjust template and lora_target parameters according to your model
16. Batch size calculation: total_batch_size = num_gpus * batch_per_gpu * gradient_accumulation_steps Training Configuration Example

### model
model_name_or_path: /path/to/mistralai/Mixtral-8x22B-Instruct-v0.1

### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: all
deepspeed: examples/deepspeed/ds_z3_offload_config.json

### dataset
dataset: custom_dataset
template: mistral
max_seq_length: 2048
max_samples: 316
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/mistral8x22b/lora/sft
logging_steps: 1
save_steps: 500
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 2
learning_rate: 1.0e-4
num_train_epochs: 1
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

Training Execution

Use the CLI tools for training, chat inference, and model merging:

llamafactory-cli train examples/train_lora/xxx.yaml
llamafactory-cli chat examples/inference/xxx.yaml
llamafactory-cli export examples/merge_lora/xxx.yaml

Recommendations:

• Test with small datasets first to validate configuration
• Monitor system disk space during data loading

Model Evaluation

After training:

1. Merge base model with LoRA weights
2. Use vLLM for faster inference
3. Ensure correct template configuration in merge script

Key Learnings

• Unsloth only supports single-GPU training
• Understand all configuration parameters before training
• Check GitHub issues first when encountering errors
• Use power-of-two GPU counts for optimal performance
• Clear cache files in ~/.cache if disk space is limited
• Be cautious with data distribution imbalances
• Test configurations with small datasets before full training
• Use temperature=0 for deterministic generation in critical applications
• Pay attention to token position - models prioritize information at the beginning
• Consider advanced techniques like PLoRA for better performance (+1-2%)
• Iterative LoRA training is generally ineffective