Introduction to Classification

Classification algorithms predict discrete categories rather than continuous values. Binary classification involves predicting between two posible outcomes, typically labeled as 0 and 1. Unlike linear regression, which can predict any numerical value, classification requires specialized algorithms like logistic regression.

Logistic Regression Fundamentals

Logistic regression uses the sigmoid function to map predictions to probabilities between 0 and 1. The sigmoid function is defined as:

g(z) = 1 / (1 + e^(-z))

where z is the linear combination of features and parameters: z = w·x + b.

Probability Interpretation

The output f(x) = g(z) represents the probability that y = 1 given input x. For example, if f(x) = 0.7, the model predicts a 70% chance of the positive class.

Decision Boundary

The decision boundary separates classes where f(x) ≥ 0.5 (predict y=1) from f(x) < 0.5 (predict y=0). For linear boundaries, this occurs when w·x + b = 0. Complex boundaries can be created using polynomial features.

Cost Function

The logistic regression cost function combines cross-entropy loss with L2 regularization:

J(w,b) = -1/m Σ [y·log(f) + (1-y)·log(1-f)] + λ/(2m) Σ w_j²

This convex function ensures gradient descent converges to the global minimum.

Gradient Descent Implementation

Parameter updates for logistic regression with regularization:

w_j := w_j - α[(1/m Σ (f(x^(i))-y^(i))x_j^(i)) + λ/m w_j]
b := b - α[1/m Σ (f(x^(i))-y^(i))]

Feature scaling accelerates convergence, similar to linear regression.

Overfitting and Regularization

Overfitting occurs when models become too complex and fit training noise. Regularization addresses this by penalizing large parameter values through the λ parameter. Strategies to combat overfitting include:

• Collecting more training data
• Feature selection
• Regularization (L2 norm)

Practical Implementation

Here's a vectorized sigmoid implementation:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

Regularized cost function calculation:

def compute_cost(X, y, w, b, lambda_):
    m = X.shape[0]
    f = sigmoid(X @ w + b)
    cost = -1/m * (y @ np.log(f) + (1-y) @ np.log(1-f))
    reg_cost = lambda_/(2*m) * (w @ w)
    return cost + reg_cost

Model Evaluation

Predictions are made by thresholding probabilities at 0.5:

def predict(X, w, b):
    return (sigmoid(X @ w + b) >= 0.5).astype(int)

Training accuracy measures the percentage of correct predictions on the training set.