1. Environment Setup and GPU Verification

Before leveraging multiple GPUs, confirm that CUDA is properly configured and all device are accessible. Darknet uses environment variables and compilation flags to manage GPU resources.

• Validate CUDA: Run nvcc --version to check the CUDA toolkit version. Use nvidia-smi to list all available GPUs and their current memory usage.

nvidia-smi  # Displays all GPUs, their IDs, and memory consumption

The output should show each GPU, such as GPU 0: Tesla V100. Note the device IDs for later use.

• Confirm Darknet Compilation: Open the Makefile and ensure the following flags are enabled:

GPU=1          # Enables GPU support
CUDNN=1        # Enables cuDNN acceleration
CUDNN_HALF=1   # Enables FP16 precision (optional)

After modification, rebuild Darknet with make -j$(nproc). The build log should mention that GPU support is active.

2. Configuration File Optimizations

Darknet uses .cfg files to define network structure and training parameters. For multi-GPU training, adjust the following core settings:

Example modifications for yolov4.cfg (located in cfg/):

[net]
batch=128          # 4 GPUs × 32 images per GPU = 128
subdivisions=16    # Each GPU processes its 32 images in 16 steps
width=608
height=608
channels=3
momentum=0.949
decay=0.0005
learning_rate=0.00261  # Scaled from 0.001 for 4 GPUs
burn_in=1000
max_batches=500500
policy=steps
steps=400000,450000
scales=0.1,0.1

Key note: Apply the linear scaling rule for the learning rate: new_LR = old_LR × (new_batch / old_batch) to prevent gradient instability.

3. Distributed Training Commands and GPU Management

Darknet's train_detector function supports multi-GPU execution. Specify GPU devices and training parameters in the command line:

./darknet detector train \
  cfg/coco.data \          # Dataset configuration
  cfg/yolov4.cfg \         # Network configuration
  yolov4.conv.137 \        # Pre-trained weights
  -gpus 0,1,2,3 \          # GPU device IDs
  -map                     # Compute mAP during training

• Device selection: Use continuous IDs (0-3) or discrete IDs (0,2,3).
• Checkpoint saving: Add -snapshot 10000 to save model states every 10,000 iterations.
• Half-precision training: Add -half to use FP16, reducing memory usage by about 30%.

4. Monitoring and Performance Tuning

Monitor load balancing and resource utilization during multi-GPU training:

• Key metrics:
  • GPU utilization: Run nvidia-smi -l 1. Ideal range is 80%–95%.
  • Memory usage: Keep per-card usage below 90% to avoid out-of-memory errors.
  • Training speed: Check the images/sec value in the log. A 4-GPU setup should achieve roughly 3.5× the speed of a single GPU.

Common issues and solutions:

Performance tips:

• When using mosaic augmentasion, set mosaic_bound=1 to reduce CPU overhead.
• If a single GPU runs out of memory, increase subdivisions to accumulate gradients across more steps.
• Use ./darknet partial to extract intermediate layer weights for faster fine-tuning.

Next Steps

These four steps provide a complete workflow for multi-GPU training with Darknet. Start with a 2-GPU configuration to test stability, then scale to 4 or more GPUs. For further improvements:

• Analyze training logs with scripts/log_parser/log_parser.py to fine-tune the learning rate schedule.
• Use the -chart parameter to generate training curves for visual monitoring.
• Explore mixed-precision training and model quantization for faster inference.