Introduction to Machine Learning

Machine learning is the discipline focused on enabling systems to learn from data rather than following explicit, rule-based programming. Algorithms analyze datasets to identify patterns and make decisions with minimal human intervention. The field is broadly categorized into supervised and unsupervised learning.

Supervised Learning

Supervised learning involves algorithms that map input features to output targets by learning from labeled examples. The model is provided with the correct answers during training, allowing it to learn the relationship between the input variables and the desired output.

• Regression: Models predict continuous numerical values, offering an infinite range of possible outputs. A classic example is forecasting real estate prices based on property size.
• Classification: Models predict discrete categories or classes, representing a finite set of possible outcomes. For instance, diagnosing a tumor as malignant or benign based on patient data.

Unsupervised Learning

Unsupervised learning algorithms operate on unlabeled data. The objective is to discover underlying structures, patterns, or groupings within the dataset without predefined categories.

• Clustering: Grouping similar data points together, such as segmenting customers into distinct market profiles based on purchasing behavior.
• Anomaly Detection: Identifying unusual data points, often used for fraud detection.
• Dimensionality Reduction: Compressing datasets into fewer features while preserving critical information.

Linear Regression Model

Linear regression establishes a linear relationship between input features and a continuous target variable. Standard notation defines x as the input feature, y as the target variable, m as the total training examples, and (x⁽ⁱ⁾, y⁽ⁱ⁾) as a specific training instance.

The model function is represented as:

f_w,b(x) = wx + b

Here, w represents the weight (slope), and b represents the bias (y-intercept). The goal is to determine the optimal values for w and b such that the prediction ŷ closely approximates the true target y.

Cost Function

The cost function quantifies the error between the model's predictions and the actual data. For linear regression, the standard metric is the Squared Error Cost Function:

J(w, b) = (1 / 2m) * Σ (f_w,b(x⁽ⁱ⁾) - y⁽ⁱ⁾)²

By computing the average of the squared differences across all training examples, the cost function evaluates how well a specific line fits the data. The objective of linear regression is to minimize this cost function. Visualizing J(w, b) often reveals a convex, bowl-shaped surface (or U-shaped curve if restricted to one parameter), where the lowest point corresponds to the optimal parameters.

Gradient Descent

Gradient descent is an optimization algorithm used to systematically find the values of w and b that minimize the cost function.

The algorithm follows these steps:

1. Initialize parameters w and b (typically to zero).
2. Continuously adjust the parameters in the direction of the steepest decrease in cost until convergence.

The update rules are:

w = w - α * ∂J/∂w

b = b - α * ∂J/∂b

The learning rate α dictates the magnitude of the steps taken. A rate that is too small results in sluggish convergence, whereas an excessively large rate may cause the algorithm to overshoot the minimum and diverge.

Gradient Descent for Linear Regression

When applying gradient descent to the squared error cost function, the partial derivatives are computed as:

∂J/∂w = (1 / m) * Σ (f_w,b(x⁽ⁱ⁾) - y⁽ⁱ⁾) * x⁽ⁱ⁾

∂J/∂b = (1 / m) * Σ (f_w,b(x⁽ⁱ⁾) - y⁽ⁱ⁾)

It is essential to update w and b simultaneously. Because the squared error cost function for linear regression is convex, gradient descent is guaranteed to converge to the global minimum. This specific variant, which calculates the error over the entire training dataset per iteration, is known as Batch Gradient Descent.

Code Implementation

The following Python implementation demonstrates calculating the cost function, computing gradients, and executing the gradient descent optimization.

import numpy as np
import copy

def calculate_squared_error(features, targets, weight, bias):
    """Computes the squared error cost function."""
    num_samples = features.shape[0]
    total_error = 0.0
    for idx in range(num_samples):
        prediction = weight * features[idx] + bias
        squared_diff = (prediction - targets[idx]) ** 2
        total_error += squared_diff
    mean_squared_error = total_error / (2 * num_samples)
    return mean_squared_error

def compute_parameter_gradients(features, targets, weight, bias):
    """Computes partial derivatives for weight and bias."""
    num_samples = features.shape[0]
    grad_weight = 0.0
    grad_bias = 0.0
    for idx in range(num_samples):
        prediction = weight * features[idx] + bias
        error = prediction - targets[idx]
        grad_weight += error * features[idx]
        grad_bias += error
    grad_weight /= num_samples
    grad_bias /= num_samples
    return grad_weight, grad_bias

def run_gradient_optimization(features, targets, init_weight, init_bias, learn_rate, iterations):
    """Executes gradient descent to find optimal weight and bias."""
    current_w = copy.deepcopy(init_weight)
    current_b = copy.deepcopy(init_bias)
    cost_history = []
    
    for step in range(iterations):
        dw, db = compute_parameter_gradients(features, targets, current_w, current_b)
        current_w -= learn_rate * dw
        current_b -= learn_rate * db
        current_cost = calculate_squared_error(features, targets, current_w, current_b)
        cost_history.append(current_cost)
        
        if step % 1000 == 0:
            print(f"Step {step}: Cost {current_cost:.4e}")
            
    return current_w, current_b, cost_history

# Dataset example
area_data = np.array([1.0, 2.0])
price_data = np.array([300.0, 500.0])

# Training parameters
initial_w = 0
initial_b = 0
alpha = 0.01
epochs = 10000

# Run optimization
optimal_w, optimal_b, _ = run_gradient_optimization(area_data, price_data, initial_w, initial_b, alpha, epochs)
print(f"Optimal parameters: weight={optimal_w:.4f}, bias={optimal_b:.4f}")