1. Algorithm Overview

Linear regression aims to minimize the mean squared error (MSE) between predictions and actual values. Gradient descent iteratively updates model parameters to reduce the cost function.

Paramter update rule:

θ := θ - α * (1/m) * Xᵀ (Xθ - y)

2. Complete MATLAB Implementation

2.1 Data Preparation

%% Generate synthetic data with noise
rng(42);               % Reproducible results
num_samples = 100;     % Number of data points
features = 2 * rand(num_samples, 1);   % Feature values in [0,2]
targets = 4 + 3 * features + randn(num_samples, 1);  % True θ₀=4, θ₁=3

%% Feature normalization (speeds up convergence)
feat_mean = mean(features);
feat_std = std(features);
features = (features - feat_mean) / feat_std;

%% Add intercept term (column of ones)
features = [ones(num_samples, 1), features];

2.2 Gradient Descent Implementation

function [params, loss_history] = gradient_descent(X, y, params, alpha, max_iter)
    m = length(y);
    loss_history = zeros(max_iter, 1);
    
    for iter = 1:max_iter
        % Compute predictions
        predictions = X * params;
        
        % Compute gradient
        grad = (1/m) * (X' * (predictions - y));
        
        % Update parameters
        params = params - alpha * grad;
        
        % Record cost
        loss_history(iter) = compute_cost(X, y, params);
        
        % Optional: adjust learning rate if loss increases
        if iter > 1 && loss_history(iter) > loss_history(iter-1)
            alpha = alpha * 0.5;
        end
    end
end

function cost = compute_cost(X, y, params)
    m = length(y);
    predictions = X * params;
    cost = (1/(2*m)) * sum((predictions - y).^2);
end

2.3 Training and Visualization

%% Initialize parameters
params = zeros(2, 1);      % [θ₀; θ₁]
learning_rate = 0.1;       % Initial learning rate
max_iterations = 500;

%% Run gradient descent
[params, loss_hist] = gradient_descent(features, targets, params, learning_rate, max_iterations);

%% Plot convergence and fit
figure;
subplot(2,1,1);
plot(1:max_iterations, loss_hist, 'b-o', 'LineWidth', 1.5);
xlabel('Iteration'); ylabel('Cost'); title('Cost Function Convergence');

subplot(2,1,2);
scatter(features(:,2), targets, 'r', 'MarkerSize', 50);
hold on;
plot(features(:,2), features * params, 'g', 'LineWidth', 2);
xlabel('Feature X'); ylabel('Target y'); title('Linear Regression Fit');
legend('Data points', 'Fitted line');

3. Extensions

3.1 Multiple Linear Regression

Extends to multiple features:

%% Generate 3-variable data
X = [ones(num_samples,1), randn(num_samples,1), 2*rand(num_samples,1)];
y = 5 + 3*X(:,2) + 4*X(:,3) + randn(num_samples,1);

%% Initialize and run
params = zeros(3,1);
[params, ~] = gradient_descent(X, y, params, 0.01, 1000);

3.2 Regularized Linear Regression (L2)

Adds a penalty term to prevent overfitting:

function cost = compute_cost_reg(X, y, params, lambda)
    m = length(y);
    preds = X * params;
    reg_term = (lambda / (2*m)) * sum(params(2:end).^2);
    cost = (1/(2*m)) * sum((preds - y).^2) + reg_term;
end

3.3 Interactive Parameter Tuning

Using MATLAB App Designer you can add sliders for learning rate, display real‑time loss curves, and a prediction input box.

4. Practical Engineering Considerations

4.1 Data Validation

%% Check for NaN values
if any(isnan(features), 'all') || any(isnan(targets))
    error('Input data contains NaN values!');
end

%% Correlation analysis
corr_matrix = corr(features(:,2:end), targets);
disp('Feature relevance analysis:');
disp(corr_matrix);

4.2 Cross‑Validation

Use k‑fold cross‑validation to evaluate generalization:

cv = cvpartition(num_samples, 'KFold', 5);
cv_errors = zeros(cv.NumTestSets, 1);

for i = 1:cv.NumTestSets
    train_idx = training(cv, i);
    test_idx = test(cv, i);
    
    params_cv = gradient_descent(features(train_idx,:), targets(train_idx), zeros(2,1), 0.1, 500);
    predictions_cv = features(test_idx,:) * params_cv;
    cv_errors(i) = mean((predictions_cv - targets(test_idx)).^2);
end

fprintf('Average cross‑validation error: %.4f\n', mean(cv_errors));