Introduction

When training deep learning models with Caffe, visualizing the training progress through loss curves is essential for monitoring model convergence. This article presents two practical approaches for generating training visualizations.

Method 1: Custom Python Script for Headless Servers

The following implementation is specifically designed for server environments without a graphical desktop. All file paths in the solver configuration must be absolute paths.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import matplotlib
matplotlib.use('Agg')
import sys
import numpy as np
import matplotlib.pyplot as plt

caffe_root = '/opt/caffe/'
sys.path.insert(0, caffe_root + 'python')
import caffe

caffe.set_mode_cpu()
solver = caffe.SGDSolver('/opt/caffe/examples/mnist/lenet_solver.prototxt')

# Configuration parameters
max_iterations = 2000
log_interval = 50
evaluation_interval = 500
evaluation_iterations = 100

# Initialize arrays for metrics
training_loss = np.zeros(int(np.ceil(max_iterations * 1.0 / log_interval)))
validation_loss = np.zeros(int(np.ceil(max_iterations * 1.0 / evaluation_interval)))
validation_accuracy = np.zeros(int(np.ceil(max_iterations * 1.0 / evaluation_interval)))

solver.step(1)
cumulative_train_loss = 0.0
cumulative_val_loss = 0.0
cumulative_accuracy = 0.0

for iteration in range(max_iterations):
    solver.step(1)
    cumulative_train_loss += solver.net.blobs['loss'].data
    
    if iteration % log_interval == 0:
        training_loss[iteration // log_interval] = cumulative_train_loss / log_interval
        cumulative_train_loss = 0.0
    
    if iteration % evaluation_interval == 0:
        for eval_iter in range(evaluation_iterations):
            solver.test_nets[0].forward()
            cumulative_val_loss += solver.test_nets[0].blobs['loss'].data
            cumulative_accuracy += solver.test_nets[0].blobs['accuracy'].data
        
        validation_loss[iteration // evaluation_interval] = cumulative_val_loss / evaluation_iterations
        validation_accuracy[iteration // evaluation_interval] = cumulative_accuracy / evaluation_iterations
        cumulative_val_loss = 0.0
        cumulative_accuracy = 0.0

# Generate visualization
plt.figure(figsize=(10, 6))
plt.plot(log_interval * np.arange(len(training_loss)), training_loss, 'g-', label='Training Loss')
plt.plot(evaluation_interval * np.arange(len(validation_loss)), validation_loss, 'y-', label='Validation Loss')
plt.plot(evaluation_interval * np.arange(len(validation_accuracy)), validation_accuracy, 'r-', label='Validation Accuracy')

plt.xlabel('Iteration')
plt.ylabel('Loss / Accuracy')
plt.legend()
plt.grid(True, alpha=0.3)
plt.savefig('/opt/caffe/examples/mnist/training_curves.png', dpi=150)
print("Training curves saved successfully")

Note: If you encounter a TypeError: 'numpy.float64' object cannot be interpreted as an index error, downgrade NumPy to version 1.11.0:

sudo pip install -U numpy==1.11.0

Method 2: Dual Axis Plotting with Matplotlib

This approach uses twinx() to display loss and accuracy on different y-axes within a single plot. Note that this method requires a desktop enviroment.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import numpy as np
import matplotlib.pyplot as plt
import sys

caffe_root = '/opt/caffe/'
sys.path.insert(0, caffe_root + 'python')
import caffe

caffe.set_mode_cpu()
solver = caffe.SGDSolver('/opt/caffe/examples/mnist/lenet_solver.prototxt')

# Training parameters
total_iterations = 2000
validation_interval = 250

# Pre-allocate arrays
training_losses = np.zeros(total_iterations)
validation_accuracies = np.zeros(int(np.ceil(total_iterations / validation_interval)))

# Main training loop
for iteration in range(total_iterations):
    solver.step(1)
    training_losses[iteration] = solver.net.blobs['loss'].data
    
    if iteration % validation_interval == 0:
        accuracy = solver.test_nets[0].blobs['accuracy'].data
        validation_accuracies[iteration // validation_interval] = accuracy
        print(f'Iteration {iteration}: Validation Accuracy = {accuracy:.4f}')

# Create dual-axis plot
fig, primary_axis = plt.subplots()
secondary_axis = primary_axis.twinx()

primary_axis.plot(np.arange(total_iterations), training_losses, 'b-', alpha=0.7, label='Training Loss')
secondary_axis.plot(
    validation_interval * np.arange(len(validation_accuracies)),
    validation_accuracies,
    'r-',
    linewidth=2,
    label='Validation Accuracy'
)

primary_axis.set_xlabel('Iteration')
primary_axis.set_ylabel('Training Loss', color='blue')
secondary_axis.set_ylabel('Validation Accuracy', color='red')
primary_axis.tick_params(axis='y', labelcolor='blue')
secondary_axis.tick_params(axis='y', labelcolor='red')

plt.title('Training Progress Visualization')
fig.legend(loc='upper right', bbox_to_anchor=(0.88, 0.88))
plt.tight_layout()
plt.show()

Method 3: Caffe's Built-in Logging Tools

Caffe provides auxiliary scripts for generating training logs and visualizations:

#!/bin/bash
# Generate timestamped log file
LOG_DIR="/opt/caffe/examples/mnist"
TIMESTAMP=$(date +%Y-%m-%d-%H-%M)
LOG_FILE="${LOG_DIR}/training_${TIMESTAMP}.log"

# Run training and capture output
cd /opt/caffe
./build/tools/caffe train --solver=examples/mnist/lenet_solver.prototxt 2>&1 | tee $LOG_FILE

Copy the required parsing scripts to your log directory:

cp /opt/caffe/tools/extra/parse_log.sh /opt/caffe/tools/extra/extract_seconds.py \
   /opt/caffe/tools/extra/plot_training_log.py.example /path/to/log/directory/

Generate the visualization plot:

python plot_training_log.py 6 training_plot.png training_log.log

Limitation: The built-in plotting tool has known accuracy issues with data processing, and certain parameter renges (4-7) may not function correctly.

Running Training in Background

For long-running training sessions via SSH, use the following command to ensure training continues even after disconnecting:

nohup python train_script.py > nohup_output.log 2>&1 &

This approach is particularly useful for extended training runs where connection stability cannot be guaranteed.