Perceptrons

A perceptron functions as a linear classifier for machine learning tasks. It can handle binary classification problems by outputting values of -1 or 1, and can be extended to multi-class classification through combinations of binary classifiers.

Convergence Issues in Perceptrons

Traditional perceptrons face limitations in convergence when dealing with non-linearly separable data, which leads to the development of more advanced neural network architectures.

Multi-Layer Perceptrons

Linear models have inherent limitations, such as assuming a linear relationship between features and outputs, which doesn't hold true in many real-world scenarios. For example, the relationship between body temperature and mortality rates, or complex image classification problems, cannot be adequately modeled with linear approaches.

To overcome these limitations, neural networks incorporate hidden layers, stacking multiple fully-connected layers together to form multi-layer perceptrons (MLPs).

Single Hidden Layer Networks

A fundamental architecture in deep learning consists of networks with a single hidden layer between input and output layers.

Activation Functions

Activation functions play several critical roles in neural networks:

• Introducing Non-linearity: Activation functions enable neural networks to learn and represent complex patterns and relationships. Without them, a neural network with any number of layers would behave like a simple linear model.
• Enhancing Model Expressiveness: Different activation functions allow neural networks to better fit training data, improving model accuracy and generalization capabilities.
• Addressing Gradient Issues: In deep networks, certain activation functions (like ReLU and its variants) help mitigate the vanishing gradient problem during backpropagation.
• Accelerating Training: Appropriate activation functions can speed up the training process. For instance, ReLU's constant gradient of 1 in the positive region facilitates faster convergence.
• Distribution Preservation: Functions like tanh and sigmoid constrain input values to specific ranges, helping maintain data distribution and preventing numerical instability from excessively large values.

Multi-Class Classification

For multi-class classification problems, neural networks typically use the softmax activation function in the output layer to produce probability distributions across multiple classes.

Summary

• Multi-layer perceptrons leverage hidden layers and activation functions to create non-linear models
• Common activation functions include Sigmoid, Tanh, and ReLU
• Softmax is typically used for multi-class classification tasks
• Key hyperparameters include the number of hidden layers and the size of each hidden layer

Implementation from Scratch

# Import necessary libraries
import numpy as np
import torch
from torch.utils import data
import torchvision
from torchvision import transforms
from d2l import torch as d2l

# Load Fashion-MNIST dataset
batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)

# Define network architecture
num_inputs = 784  # Fashion-MNIST images are 28x28 pixels
num_outputs = 10  # 10 classes for Fashion-MNIST
num_hiddens = 256  # Number of neurons in the hidden layer

# Initialize parameters
weight1 = torch.randn(num_inputs, num_hiddens, requires_grad=True) * 0.01
bias1 = torch.zeros(num_hiddens, requires_grad=True)
weight2 = torch.randn(num_hiddens, num_outputs, requires_grad=True) * 0.01
bias2 = torch.zeros(num_outputs, requires_grad=True)

# Collect parameters
parameters = [weight1, bias1, weight2, bias2]

# Implement ReLU activation function
def relu_activation(input_tensor):
    return torch.max(input_tensor, torch.zeros_like(input_tensor))

# Define the neural network model
def neural_network_model(X):
    # Flatten input
    X = X.reshape((-1, num_inputs))
    # First hidden layer with ReLU activation
    hidden_output = relu_activation(X @ weight1 + bias1)
    # Output layer
    return hidden_output @ weight2 + bias2

# Set up loss function
loss_function = torch.nn.CrossEntropyLoss()

Training Evaluation Functions

# Function to calculate accuracy
def calculate_accuracy(predictions, actual_labels):
    if len(predictions.shape) > 1 and predictions.shape[1] > 1:
        predictions = predictions.argmax(axis=1)
    comparison = predictions.type(actual_labels.dtype) == actual_labels
    return float(comparison.type(actual_labels.dtype).sum())

# Helper class for accumulating metrics during training
class MetricsAccumulator:
    def __init__(self, num_metrics):
        self.metrics_data = [0.0] * num_metrics
    
    def add(self, *args):
        self.metrics_data = [a + float(b) for a, b in zip(self.metrics_data, args)]
    
    def reset(self):
        self.metrics_data = [0.0] * len(self.metrics_data)
    
    def __getitem__(self, index):
        return self.metrics_data[index]

# Function to evaluate model accuracy on a dataset
def evaluate_model_accuracy(model, data_loader):
    if isinstance(model, torch.nn.Module):
        model.eval()  # Set model to evaluation mode
    
    metrics = MetricsAccumulator(2)  # Track correct predictions and total samples
    
    with torch.no_grad():
        for inputs, labels in data_loader:
            metrics.add(calculate_accuracy(model(inputs), labels), labels.numel())
    
    return metrics[0] / metrics[1]

# Function to train for one epoch
def train_one_epoch(model, train_loader, loss_fn, optimizer):
    if isinstance(model, torch.nn.Module):
        model.train()  # Set model to training mode
    
    metrics = MetricsAccumulator(3)  # Track loss, accuracy, and sample count
    
    for inputs, labels in train_loader:
        predictions = model(inputs)
        loss = loss_fn(predictions, labels)
        
        if isinstance(optimizer, torch.optim.Optimizer):
            optimizer.zero_grad()
            loss.mean().backward()
            optimizer.step()
        else:
            loss.sum().backward()
            optimizer(inputs.shape[0])
        
        metrics.add(float(loss.sum()), calculate_accuracy(predictions, labels), labels.numel())
    
    return metrics[0] / metrics[2], metrics[1] / metrics[2]

Training Process

# Set training parameters
epochs = 10
learning_rate = 0.01

# Create optimizer
optimizer = torch.optim.SGD(parameters, lr=learning_rate)

# Train the model
for epoch in range(epochs):
    train_loss, train_accuracy = train_one_epoch(neural_network_model, train_iter, loss_function, optimizer)
    test_accuracy = evaluate_model_accuracy(neural_network_model, test_iter)
    
    print(f'Epoch {epoch+1}/{epochs}, Training Loss: {train_loss:.4f}, '
          f'Training Accuracy: {train_accuracy:.4f}, Test Accuracy: {test_accuracy:.4f}')

Concise Implementation with PyTorch

# Import PyTorch modules
import torch
from torch import nn
from d2l import torch as d2l

# Load dataset
batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)

# Define network architecture
network = nn.Sequential(
    nn.Flatten(),  # Flatten input images
    nn.Linear(784, 256),  # First hidden layer
    nn.ReLU(),  # Activation function
    nn.Linear(256, 10)  # Output layer
)

# Initialize weights
def initialize_weights(layer):
    if type(layer) == nn.Linear:
        nn.init.normal_(layer.weight, std=0.01)

network.apply(initialize_weights)

# Set up loss function and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(network.parameters(), lr=0.1)

# Train the model
epochs = 10
for epoch in range(epochs):
    network.train()  # Set model to training mode
    total_loss, total_correct, total_samples = 0, 0, 0
    
    for inputs, labels in train_iter:
        # Forward pass
        outputs = network(inputs)
        loss = criterion(outputs, labels)
        
        # Backward pass and optimization
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        # Update metrics
        total_loss += loss.item() * labels.size(0)
        total_correct += (outputs.argmax(1) == labels).sum().item()
        total_samples += labels.size(0)
    
    # Calculate metrics
    train_loss = total_loss / total_samples
    train_accuracy = total_correct / total_samples
    
    # Evaluate on test set
    network.eval()  # Set model to evaluation mode
    test_correct, test_samples = 0, 0
    
    with torch.no_grad():
        for inputs, labels in test_iter:
            outputs = network(inputs)
            test_correct += (outputs.argmax(1) == labels).sum().item()
            test_samples += labels.size(0)
    
    test_accuracy = test_correct / test_samples
    
    print(f'Epoch {epoch+1}/{epochs}, Loss: {train_loss:.4f}, '
          f'Train Accuracy: {train_accuracy:.4f}, Test Accuracy: {test_accuracy:.4f}')