PyTorch Model Training Demo Code

In PyTorch, model training typically involves several key steps: defining the model, defining the loss function, selecting an optimizer, preparing a data loader, and writing the training loop. Below is a simple PyTorch model training demo code that implements a basic neural network for handwritten digit recognition (using the MNIST dataset).

First, ensure you have installed PyTorch and torchvision (for loading the MNIST dataset).

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Define the model
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(28*28, 500)  # Input layer to hidden layer
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(500, 10)     # Hidden layer to output layer

    def forward(self, x):
        x = x.view(-1, 28*28)  # Flatten the image
        x = self.relu(self.fc1(x))
        x = self.fc2(x)
        return x

# Prepare data
transform = transforms.Compose([
    transforms.ToTensor(),  # Convert to Tensor
    transforms.Normalize((0.5,), (0.5,))  # Normalize
])

train_dataset = datasets.MNIST(root='./data', train=True, transform=transform, download=True)
train_loader = DataLoader(dataset=train_dataset, batch_size=64, shuffle=True)

# Instantiate model, loss function, and optimizer
model = Net()
criterion = nn.CrossEntropyLoss()  # Multi-class cross-entropy loss
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

# Train the model
num_epochs = 5
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward pass and optimization
        optimizer.zero_grad()  # Zero the gradients
        loss.backward()        # Backpropagation
        optimizer.step()       # Update parameters

        if (i+1) % 100 == 0:
            print(f'Epoch [{epoch+1}/{num_epochs}], Step [{i+1}/{len(train_loader)}], Loss: {loss.item():.4f}')

print('Finished Training')

In this example, we defined a simple fully connected neural network Net that contains two fully connected layers and a ReLU activation function. We used the MNIST dataset for training, which contains images of handwritten digits. We used the cross-entropy loss functon and the SGD optimizer.

In the training loop, we iterate over each batch in the data loader, perform a forward pass for each batch to calculate the loss, then perform a backward pass to calculate gradients, and use the optimizer to update the model's parameters. Every 100 batches, we print the current loss value to monitor the training process.

Dataset & DataLoader

Three essential functions in Dataset: __init__, __getitem__, __len__.

The timing of calling __getitem__ is when iterating through the DataLoader. When iterating through the DataLoader, the returned attributes are the values returned by the __getitem__ method.

Executing the code below shows the calling timing of __getitem__.

By changing num_workers, you can see that data is indeed being fetched by multiple threads.

import os

from torch.utils.data import Dataset, DataLoader


class MyDataset(Dataset):
    def __init__(self):
        print('init my dataset')
        self.imgs = os.listdir('../YOLO5/datasets/coco128/images/train2017')

    def __getitem__(self, idx):
        print('get item')
        return self.imgs[idx]

    def __len__(self):
        return len(self.imgs)


if __name__ == '__main__':
    my_dataset = MyDataset()
    print('create my_dataloader')
    my_dataloader = DataLoader(dataset=my_dataset, batch_size=8, shuffle=True, num_workers=0, drop_last=True)
    for idx, data in enumerate(my_dataloader):
        print(idx, data)

Below, we look at the actual application of Dataset and DataLoader using code from YOLOv5. It shows the values returned in __getitem__, and then when iterating through pbar (the DataLoader), it obtains attributes like imgs, targets, paths.

Note that when using enumerate, the index is also returned.

def __getitem__(self, index):
    # Omitted thousands of lines of code
    return torch.from_numpy(img), labels_out, self.im_files[index], shapes

pbar = enumerate(train_loader)
LOGGER.info(("\n" + "%11s" * 7) % ("Epoch", "GPU_mem", "box_loss", "obj_loss", "cls_loss", "Instances", "Size"))
if RANK in {-1, 0}:
    pbar = tqdm(pbar, total=nb, bar_format=TQDM_BAR_FORMAT)  # progress bar
optimizer.zero_grad()
for i, (imgs, targets, paths, _) in pbar:  # batch -------------------------------------------------------------
    callbacks.run("on_train_batch_start")

Network Architecture Visualization Tool

• Visualization project: Netron
• It is recommended to use the ONNX format for more detailed model visualization.

Principles of Deep Learning

The principles of deep learning are primarily based on the complex structure of Artificial Neural Networks (ANNs), especially advanced variants like Multi-Layer Perceptrons (MLPs), Convolutional Neural Networks (CNNs), and Recurrent Neural Networks (RNNs). These networks learn representations and patterns from data by mimicking the connections and communication methods between neurons in the human brain. Here are several key aspects of deep learning principles:

1. Basics of Artificial Neural Networks:

   • Neurons: The basic unit of a neural network, simulating biological neurons. Each neuron receives input signals from other neurons, performs a weighted sum (including a bias term), and then processes through an activation function (such as Sigmoid, ReLU, etc.) to produce an output signal.
   • Layers: Multiple neurons are arranged into layers. The input layer receives raw data, hidden layers process the data, and the output layer produces the final result.
   • Weights and Biases: The strength of connections between neurons is represented by weights, and bias terms are used to adjust the input to the activation function.

2. Forward Propagation: During training, input data is propagated forward through the network layer by layer until the output layer produces a prediction. During this process, the network's weights and biases remain unchanged.

3. Loss Function: Used to quantify the difference between the predicted result and the true result. Common loss functions include Mean Squared Error (MSE), Cross-Entropy Loss, etc.

4. Backward Propagation: Based on the gradient of the loss function, gradients are propagated backward layer by layer using the chain rule, updating the weights and biases of each layer's neurons to minimize the loss function. This is achieved through gradient descent (or its variants like Adam, RMSprop, etc.) algorithms.

5. Optimization Algorithms: Algorithms used to update the network's parameters, aiming to find parameter values that minimize the loss function. Gradient descent is one of the most commonly used optimization algorithms, but modern deep learning frameworks offer a variety of optimizers to choose from.

6. Overfitting and Regularization: Overfitting occurs when a model performs well on training data but poorly on unseen data. To mitigate overfitting, regularization techniques (such as L1/L2 regularization, Dropout, etc.) can be used to reduce the model's complexity.

7. Activation Functions: Activation functions introduce non-linearity, enabling neural networks to learn and represent complex non-linear relationships. Different activation functions are suitable for different tasks and data.

8. Convolutional Neural Networks (CNNs): Particularly suitable for processing image data, using structures like convolutional layers and pooling layers to automatically extract image features.

9. Recurrent Neural Networks (RNNs) and their variants (like LSTM, GRU): Particularly suitable for processing sequential data (such as text, time series), capable of capturing temporal dependencies in the data.

The succsess of deep learning is largely due to the clever combination of these components, the availability of large-scale data, and significant improvements in computing power. Through these techniques, deep learning models can achieve performance levels that match or even exceed human capabilities on many complex tasks.

The Role of Multiple Layers in Convolutional Networks

It is often said that features progress "from low-level edges, colors, textures to mid-level simple shapes, patterns, and then to high-level complex structures and semantic information." How can this be visualized? Has anyone visualized this before? Where can it be found?

CNNs require multiple layers mainly for the following reasons:

1. Feature Abstraction and Combination

• Layer-by-layer feature extraction: CNNs extract image features layer by layer through multiple convolution operations. From low-level features like edges, colors, and textures, to mid-level features like simple shapes and patterns, and then to high-level features like complex structures and semantic information, this process is gradual. Each layer further abstracts and combines features based on the previous layer, thereby building more complex and discriminative feature representations.
• Combining features from different levels: The multi-layer structure allows CNNs to combine features from different levels, enabling comprehensive understanding and modeling of input data. This combination capability is particularly important for processing complex image data, as information in images is often distributed across different scales and levels.

2. Improving Model Expressiveness

• Increasing non-linearity: A non-linear activation function (e.g., ReLU) is usually applied after each convolution, increasing the model's non-linear expressiveness. Multiple layers of non-linear transformation can fit more complex functional relationships, allowing the model to learn complex patterns in the input data.
• Avoiding information loss: Although multiple convolution and pooling operations reduce the resolution of feature maps to some extent, the multi-layer structure can retain more information, preventing information loss caused by excessive dimensionality reduction at a single level.

3. Expanding Receptive Field

• Accumulation of receptive fields: In CNNs, the receptive field refers to the area of the input image that a convolution kernel can 'see'. As the number of convolutional layers increases, the receptive field of each layer's neurons gradually expands, enabling the capture of more global and abstract features. This cumulative effect of receptive fields is a key factor in CNNs' ability to handle complex image tasks.

4. Adapting to Features at Different Scales

• Multi-scale feature extraction: By using convolution kernels of different sizes and strides, CNNs can extract features at different scales. The multi-layer structure allows CNNs to process features at multiple scales simultaneously, leading to a more comprehensive understanding of the input data.

5. Improving Model Robustness

• Feature redundancy: The multi-layer structure allows CNNs to learn multiple feature representations that are redundant and complementary. When some features are affected by noise or occlusion, other features can still provide effective information, thereby improving the model's robustness.

In summary, CNNs require multiple layers primarily for layer-by-layer feature extraction and combination, improving model expressiveness, expanding the receptive field, adapting to features at different scales, and improving model robustness. These advantages enable CNNs to achieve significant results in tasks such as image recognition, object detection, and semantic segmentation.

The Role of Convolutional Layers in Deep Learning

The convolution layer in deep learning is the core component of Convolutional Neural Networks (CNNs). It performs convolution operations on input data, which is especially crucial in image processing and computer vision tasks. The main roles of convolutional layers can be summarized as follows:

1. Feature Extraction: Convolutional layers perform sliding window dot products on input data using a set of learnable filters (also called convolution kernels or weights), thereby extracting local features from the input data. These features can be edges, corners, textures, etc., and are critical for subsequent image recognition or classification tasks.

2. Reducing the Number of Parameters: Because each neuron in a convolutional layer is connected to only a local region of the previous layer (local connectivity), and this connection weight (the convolution kernel) is shared across different positions in the input data (weight sharing), the number of parameters the model needs to learn is significantly reduced. This reduces model complexity, improves training efficiency, and decreases the risk of overfitting.

3. Translation Invariance: Due to the weight sharing property in convolutional layers, the network has a degree of invariance to translation transformations (i.e., changes in the position of an object in the image) in the input data. This means that no matter where an object appears in the image, the network can effectively recognize it.

4. Dimensionality Reduction and Pooling: Although convolutional layers themselves do not directly perform dimensionality reduction, they are typically used in conjunction with pooling layers. Pooling layers downsample the output of convolutional layers (e.g., max pooling, average pooling), further reducing the spatial size of the data (dimensionality reduction), extracting more important features, and increasing the model's robustness to small deformations (e.g., rotation, scaling) in the input data.

5. Learning Hierarchical Features: By stacking multiple convolutional layers, the network can learn hierarchical feature representations from simple to complex. Shallow convolutional layers typically learn low-level features like edges and lines, while deeper convolutional layers can learn more advanced, abstract features that are crucial for completing complex image recognition or classification tasks.

In summary, convolutional layers play a vital role in deep learning, especially when processing image data. They provide strong support for subsequent tasks like image recognition, classification, and detection through feature extraction, parameter reduction, translation invariance, dimensionality reduction/pooling, and learning hierarchical features.

Encoder-Decoder Architecture

The Encoder-Decoder structure in deep learning is a widely used network architecture in fields such as image processing and natural language processing (NLP). This structure effectively combines two processes: feature extraction (encoding) and result generation (decoding). The following is a detailed analysis of this structure.

I. Encoder

Definition and Function:

• The encoder is responsible for extracting useful feature information from the input data. These features are typically abstract representations of different objects or regions in the input data, which aid subsequent processing or generation tasks.

Structure and Operations:

• The encoder usually consists of a series of network layers, such as convolutional layers, pooling layers, etc.
• In NLP, the encoder (e.g., in Transformer) processes an input sequence and outputs a sequence of hidden states (contextual representations).