Linear Regression Fundamentals The linear model is defined as: $y = Xw + b + \epsilon$, where $w$ represents weights and $b$ is the bias term. Model evaluation relies on loss functions that quantify prediction errors:

MSE Loss Function: $$ l^{(i)}(w,b) = \frac{1}{2}(\hat{y}^{(i)} - y^{(i)})^2 \ L(w,b) = \frac{1}{n}\sum_{i=1}^{n}l^{(i)}(w,b) $$

Parameter Optimization: Gradient descent updates parameters along the negatvie gradient direction: $$ (w,b) \leftarrow (w,b) - \lambda\frac{\sum_{i\in{B}}\partial_{(w,b)}l^{(i)}(w,b)}{|B|} $$

Key hyperparameters include learning rate ($\lambda$) and batch size. Smaller batches introduce noise that can improve generalization but require careful tuning.

Implemantation Workflow

Data Preparation: Custom dataset loader with transformations Model Definition:

class LinearModel(nn.Module):
    def __init__(self, input_dim):
        super().__init__()
        self.linear = nn.Linear(input_dim, 1)
    
    def forward(self, x):
        return self.linear(x)

Loss Function: criterion = nn.MSELoss()Optimizer: optimizer = torch.optim.SGD(model.parameters(), lr=0.01)Training Loop:

for epoch in range(epochs):
    for inputs, targets in dataloader:
        outputs = model(inputs)
        loss = criterion(outputs, targets)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Softmax Regression for Multi-class Classification Architecture: Fully connected layer + softmax activation $$ \hat{y}i = \frac{e^{o_i}}{\sum{k}e^{o_k}} $$

Cross-Entropy Loss: $$ l(y,\hat{y}) = -\sum_{j=1}^{q}y_{j}\log\hat{y}_j $$

Information Theory Concepts:

Entropy: $H(P) = -\sum p_i \log p_i$ KL Divergence: $D_{KL}(P||Q) = \sum p_i (\log \frac{p_i}{q_i})$ Optimization Insight: Minimizing cross-entropy aligns predicted probabilities with true labels

Implementation Example

class SoftmaxClassifier(nn.Module):
    def __init__(self, input_size, num_classes):
        super().__init__()
        self.fc = nn.Linear(input_size, num_classes)
    
    def forward(self, x):
        return F.softmax(self.fc(x), dim=1)

# Training loop with cross-entropy
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

for epoch in range(num_epochs):
    for images, labels in train_loader:
        outputs = model(images)
        loss = criterion(outputs, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()