Dropout serves as a widely adopted regularization method to mitigate overfitting in deep neural networks, complementing techniques like weight decay. The specific variant discussed here is inverted dropout. This mechanism involves randomly setting a portion of the hidden unit activations to zero during the training phase.

Mathematical Foundation

Consider a multi-layer perceptron where the hidden layer units $h$ are computed from inputs $x$ via weights $w$, biases $b$, and an activation function $ ho$:

$$ h = \rho(x_1 w_1 + x_2 w_2 + \dots + b) $$

When applying dropout with probability $p$ (the drop rate), each unit $h_i$ has a probability $p$ of being forced to zero. Conversely, with probability $1-p$, the value is scaled up by dividing by $(1-p)$ to preserve the expected magnitude.

Let $\xi_i$ be a Bernoulli random variable where $P(\xi_i=0)=p$ and $P(\xi_i=1)=1-p$. The output $h'_i$ after dropout is defined as:

$$ h'_i = \frac{\xi_i}{1-p} h_i $$

Taking the expectation:

$$ E(h'_i) = \frac{E(\xi_i)}{1-p} h_i = \frac{1-p}{1-p} h_i = h_i $$

Thus, the mean value of the hidden units remains unchanged despite the stochastic process. During inference (testing), dropout is disabled entirely to ensure deterministic outputs.

Manual Implementation from Scratch

The following Python/PyTorch implementation creates a custom inversion function.

import torch
import numpy as np

def apply_inverted_dropout(data_tensor, drop_rate):
    """
    Applies inverted dropout to the input tensor.
    
    Args:
        data_tensor: Input tensor to be modified.
        drop_rate: Probability of dropping a unit (0 to 1).
    """
    keep_ratio = 1.0 - drop_rate
    if keep_ratio == 0:
        return torch.zeros_like(data_tensor)
    
    # Generate mask based on random distribution
    # Logic: If random value > drop_rate, keep it (1), else discard (0)
    mask = (torch.rand(size=data_tensor.shape) > drop_rate).float()
    
    # Scale kept values to maintain expectation
    return (data_tensor * mask) / keep_ratio

Testing the logic with diffeernt thresholds:

# Create sample input data
input_data = torch.arange(16).reshape(2, 8)

# Test scenarios
print("No Drop:", apply_inverted_dropout(input_data, 0.0))
print("Half Drop:", apply_inverted_dropout(input_data, 0.5))
print("Full Drop:", apply_inverted_dropout(input_data, 1.0))

Model Configuration

We construct a Multi-Layer Perceptron (MLP) tailored for image classification tasks (similar to Fashion-MNIST). Two hidden layers are utilized, each with 256 neurons.

# Hyperparameters
input_dim = 784
output_dim = 10
hidden_dims = [256, 256]

# Initialize parameters manually
weight_layer_1 = torch.tensor(np.random.normal(0, 0.01, (input_dim, hidden_dims[0])), dtype=torch.float32, requires_grad=True)
bias_layer_1 = torch.zeros(hidden_dims[0], requires_grad=True)

weight_layer_2 = torch.tensor(np.random.normal(0, 0.01, (hidden_dims[0], hidden_dims[1])), dtype=torch.float32, requires_grad=True)
bias_layer_2 = torch.zeros(hidden_dims[1], requires_grad=True)

weight_output = torch.tensor(np.random.normal(0, 0.01, (hidden_dims[1], output_dim)), dtype=torch.float32, requires_grad=True)
bias_output = torch.zeros(output_dim, requires_grad=True)

params = [weight_layer_1, bias_layer_1, weight_layer_2, bias_layer_2, weight_output, bias_output]

Training Loop and Evaluation

The forward pass applies ReLU activation followed by conditional dropout. Dropout rates typically increase towards deeper layers (e.g., 0.2 then 0.5). The evaluation function must explicitly switch the model to evaluation mode to disable dropout logic.

drop_probs = [0.2, 0.5]

def predict_model(inputs, training_phase=True):
    # Reshape inputs
    x = inputs.view(-1, input_dim)
    
    # Layer 1
    h1 = torch.matmul(x, weight_layer_1) + bias_layer_1
    h1 = torch.nn.functional.relu(h1)
    
    if training_phase:
        h1 = apply_inverted_dropout(h1, drop_probs[0])
    
    # Layer 2
    h2 = torch.matmul(h1, weight_layer_2) + bias_layer_2
    h2 = torch.nn.functional.relu(h2)
    
    if training_phase:
        h2 = apply_inverted_dropout(h2, drop_probs[1])
    
    # Output
    logits = torch.matmul(h2, weight_output) + bias_output
    return logits

# Custom accuracy calculation ensuring correct mode handling
def compute_accuracy(data_loader, network_func):
    total_correct = 0
    total_samples = 0
    
    for x_batch, y_batch in data_loader:
        # Ensure inference mode during evaluation
        predictions = network_func(x_batch, training_phase=False).argmax(dim=1)
        matches = (predictions == y_batch).sum().item()
        total_correct += matches
        total_samples += y_batch.shape[0]
        
    return total_correct / total_samples

Standard PyTorch Approach

Rather than manual parameter management, standard library layers offer optimized implementations. nn.Dropout automatically handles scaling and masking without modifying code structure significantly.

from torch import nn

class SimpleDropoutNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dims[0])
        self.relu1 = nn.ReLU()
        self.dropout1 = nn.Dropout(drop_probs[0])
        
        self.fc2 = nn.Linear(hidden_dims[0], hidden_dims[1])
        self.relu2 = nn.ReLU()
        self.dropout2 = nn.Dropout(drop_probs[1])
        
        self.out_layer = nn.Linear(hidden_dims[1], output_dim)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu1(x)
        x = self.dropout1(x)
        
        x = self.fc2(x)
        x = self.relu2(x)
        x = self.dropout2(x)
        
        return self.out_layer(x)

model_instance = SimpleDropoutNet()
optimizer = torch.optim.SGD(model_instance.parameters(), lr=0.5)

During execution, enabling model.train() activates the dropout mechanisms, while model.eval() disables them, returning consistent results suitable for testing.