Environment Setup

Download Source Code

Obtain the official YOLOv8 repository from the GitHub project page.

Install Required Dependencies

Configure PyTorch environment following standard installation procedures available online.

Prepare Your Dataset

This example uses a fruit detecsion dataset. The directory structure should follow this pattern:

• Root Directory
  • custom_dataset (user-created folder containing the dataset)
    • annotations (contains XML annotation files)
    • images (contains image files)
    • splits
      • main (generated by running split script, creates train.txt, val.txt, test.txt, trainval.txt)
  • partition_script.py (splits dataset into training, validation, and test sets)
  • converter_script.py (generates paths for training, validation, and test sets)
  • dataset_config.yaml (dataset and class definitions)
• training_script.py (execution file)

The annotations directory contains XML files, while the images directory holds PNG format image files.

Execute Partition Script

The splits/main subdirectory will contain train.txt, val.txt, test.txt, and trainval.txt files generated by the partition script.

# Import required libraries
import os
import random
import argparse

# Create argument parser
parser = argparse.ArgumentParser()

# Add command line arguments for XML path
parser.add_argument('--xml_source', default='annotations', type=str, help='input xml label path')

# Add command line arguments for output path
parser.add_argument('--output_dir', default='splits/main', type=str, help='output txt label path')

# Parse arguments
args = parser.parse_args()

# Define split ratios
validation_total_ratio = 0.9
train_validation_ratio = 8/9

xml_directory = args.xml_source
txt_output = args.output_dir
total_annotations = os.listdir(xml_directory)

if not os.path.exists(txt_output):
    os.makedirs(txt_output)

annotation_count = len(total_annotations)
index_list = range(annotation_count)

validation_size = int(annotation_count * validation_total_ratio)
training_size = int(validation_size * train_validation_ratio)

validation_indices = random.sample(index_list, validation_size)
training_indices = random.sample(validation_indices, training_size)

trainval_file = open(txt_output + '/trainval.txt', 'w')
test_file = open(txt_output + '/test.txt', 'w')
train_file = open(txt_output + '/train.txt', 'w')
val_file = open(txt_output + '/val.txt', 'w')

for idx in index_list:
    name = total_annotations[idx][:-4] + '\n'
    if idx in validation_indices:
        trainval_file.write(name)
        if idx in training_indices:
            train_file.write(name)
        else:
            val_file.write(name)
    else:
        test_file.write(name)

trainval_file.close()
train_file.close()
val_file.close()
test_file.close()

After running the partition script, you'll generate train.txt, val.txt, test.txt, and trainval.txt files.

Execute Conversion Script

Run the conversion script to generate training, test, and validation file lists. The script location is shown below:

# -*- coding:utf-8 -*-

# Import necessary libraries
import xml.etree.ElementTree as ET
import os
from os import getcwd

# Define dataset splits
datasets = ['train', 'val', 'test']

# Modify these categories according to your dataset
# categories = ['Banana', 'Snake fruit', 'Dragon fruit', 'Pineapple']
categories = ["banana", "snake fruit", "dragon fruit", "pineapple"]

absolute_path = os.getcwd()
print(absolute_path)

def normalize_coordinates(size, box):
    width_reciprocal = 1./(size[0])
    height_reciprocal = 1./(size[1])
    center_x = (box[0] + box[1]) / 2.0 - 1
    center_y = (box[2] + box[3]) / 2.0 - 1
    bbox_width = box[1] - box[0]
    bbox_height = box[3] - box[2]
    center_x = center_x * width_reciprocal
    bbox_width = bbox_width * width_reciprocal
    center_y = center_y * height_reciprocal
    bbox_height = bbox_height * height_reciprocal
    return center_x, center_y, bbox_width, bbox_height

def process_annotation(image_identifier):
    annotation_file = open('./annotations/%s.xml' % (image_identifier), encoding='UTF-8')
    output_file = open('./labels/%s.txt' % (image_identifier), 'w')
    tree = ET.parse(annotation_file)
    root = tree.getroot()

    filename = root.find('filename').text
    extension = filename.split(".")[1]

    size_info = root.find('size')
    img_width = int(size_info.find('width').text)
    img_height = int(size_info.find('height').text)
    
    for element in root.iter('object'):
        difficulty_level = element.find('difficult').text
        class_name = element.find('name').text
        if class_name not in categories or int(difficulty_level) == 1:
            continue
        category_id = categories.index(class_name)
        bounding_box = element.find('bndbox')
        coordinates = (float(bounding_box.find('xmin').text), float(bounding_box.find('xmax').text), 
                      float(bounding_box.find('ymin').text), float(bounding_box.find('ymax').text))
        x_min, x_max, y_min, y_max = coordinates
        
        if x_max > img_width:
            x_max = img_width
        if y_max > img_height:
            y_max = img_height
        
        coordinates = (x_min, x_max, y_min, y_max)
        normalized_coords = normalize_coordinates((img_width, img_height), coordinates)
        output_file.write(str(category_id) + " " + " ".join([str(coord) for coord in normalized_coords]) + '\n')
    
    return extension

working_dir = getcwd()

for split in datasets:
    if not os.path.exists('./labels/'):
        os.makedirs('./labels/')
    image_identifiers = open('./splits/main/%s.txt' % (split)).read().strip().split()
    list_output = open('./%s.txt' % (split), 'w')
    for identifier in image_identifiers:
        file_extension = process_annotation(identifier)
        list_output.write(absolute_path + '/images/%s.%s\n' % (identifier, file_extension))
    list_output.close()

Running the convertion script generates train.txt, test.txt, and val.txt files with proper image paths.

Create Configuration File

Create a YAML configuration file (e.g., dataset_config.yaml) with the following content:

train: E:\\custom_project\\yolo_training\\custom_dataset\\train.txt
val: E:\\custom_project\\yolo_training\\custom_dataset\\val.txt

nc: 4

names: ["banana", "snake fruit", "dragon fruit", "pineapple"]

Training Methods

Two approaches for training with custom datasets:

Command Line Execution

Command template:

yolo task=detect      mode=train       model=yolov8n.pt           args...
      classify         predict           yolov8n-cls.yaml     args...
      segment          val               yolov8n-seg.yaml     args...
                     export            yolov8n.pt           format=onnx   args...

Execute the following command in you're project directory:

yolo task=detect mode=train model=yolov8s.yaml data=E:\custom_project\yolo_training\custom_dataset\dataset_config.yaml epochs=50 batch=4

Python Script Execution

Create a training script (training_script.py):

from ultralytics import YOLO

if __name__ == '__main__':
    # Initialize model
    model = YOLO("yolov8s.yaml")

    # Train the model
    model.train(data=r"E:\custom_project\yolo_training\custom_dataset\dataset_config.yaml",
                seed=0,
                epochs=200,
                batch=4,
                workers=2)

Training Parameters (default.yaml)

Key training parameters are stored in the default.yaml file within the ultralytics package. Here's the complete configuration:

# Default training settings and hyperparameters

task: detect
mode: train

# Training Settings
model:
data: ./custom_dataset/dataset_config.yaml
epochs: 100
patience: 50
batch: 1
imgsz: 640
save: True
save_period: -1
cache: False
device: 0
workers: 4
project:
name:
exist_ok: False
pretrained: True
optimizer: auto
verbose: True
seed: 0
deterministic: True
single_cls: False
rect: False
cos_lr: False
close_mosaic: 10
resume: False
amp: True
fraction: 1.0
profile: False
freeze: None

# Validation Settings
val: True
split: val
save_json: False
save_hybrid: False
conf:
iou: 0.7
max_det: 300
half: False
dnn: False
plots: True

# Prediction Settings
source:
show: False
save_txt: False
save_conf: False
save_crop: False
show_labels: True
show_conf: True
vid_stride: 1
stream_buffer: False
line_width:
visualize: False
augment: False
agnostic_nms: False
classes:
retina_masks: False
boxes: True

# Export Settings
format: torchscript
keras: False
optimize: False
int8: False
dynamic: False
simplify: False
opset:
workspace: 4
nms: False

# Hyperparameters
lro: 0.01
lrf: 0.01
momentum: 0.937
weight_decay: 0.0005
warmup_epochs: 3.0
warmup_momentum: 0.8
warmup_bias_lr: 0.1
box: 7.5
cls: 0.5
dfl: 1.5
pose: 12.0
kobj: 1.0
label_smoothing: 0.0
nbs: 64
hsv_h: 0.015
hsv_s: 0.7
hsv_v: 0.4
degrees: 0.0
translate: 0.1
scale: 0.5
shear: 0.0
perspective: 0.0
flipud: 0.0
fliplr: 0.5
mosaic: 1.0
mixup: 0.0
copy_paste: 0.0

# Custom Config
cfg:

# Tracker Settings
tracker: botsort.yaml