Linear regression is a fundamental approach for modeling the relationship between a scalar response and one or more explanatory variables. This implementation explores both simple (univariate) and multiple (multivariate) linear regression using NumPy and Pandas.

Univariate Linear Regression

Simple linear regression involves a single independent variable. The goal is to find a linear function that predicts the dependent variable as accurately as possible.

Data Preparation and Visualization

First, we load the dataset and visualize the relationship between the feautres.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Loading dataset
df = pd.read_csv('ex1data1.txt', names=['Population', 'Profit'])

# Adding a column of ones for the intercept term
df.insert(0, 'Ones', 1)

# Exploratory Data Analysis
df.plot.scatter('Population', 'Profit', alpha=0.5)
plt.title('Profit vs Population')
plt.show()

# Extracting features and target
X_matrix = df.iloc[:, :-1].values
y_vector = df.iloc[:, -1].values.reshape(-1, 1)

Analytical Solution: The Normal Equation

The Normal Equation provides an analytical way to solve for the optimal parameters without iteration.

def solve_normal_equation(X, y):
    return np.linalg.inv(X.T @ X) @ X.T @ y

weights_analytical = solve_normal_equation(X_matrix, y_vector)
print(f'Optimal weights (Normal Equation): {weights_analytical.flatten()}')

Iterative Solution: Gradient Descent

Gradient descent minimizes the cost function by iteratively updating the parameters.

def compute_mse_cost(X, y, weights):
    error = (X @ weights) - y
    return np.sum(np.square(error)) / (2 * len(X))

def run_gradient_descent(X, y, weights, lr, iterations):
    history = []
    m = len(X)
    
    for _ in range(iterations):
        gradient = (X.T @ (X @ weights - y)) / m
        weights = weights - lr * gradient
        history.append(compute_mse_cost(X, y, weights))
        
    return weights, history

# Hyperparameters
initial_weights = np.zeros((2, 1))
learning_rate = 0.01
epochs = 1500

final_weights, cost_history = run_gradient_descent(X_matrix, y_vector, initial_weights, learning_rate, epochs)

# Visualizing Cost Convergence
plt.plot(range(epochs), cost_history)
plt.xlabel('Iterations')
plt.ylabel('Cost (MSE)')
plt.title('Cost Reduction over Time')
plt.show()

Model Prediction

Once the weight are optimized, we can predict outcomes for new data points.

def predict_value(input_x, weights):
    features = np.array([1, input_x])
    return features @ weights

val = float(input("Enter population value for prediction: "))
result = predict_value(val, final_weights)
print(f'Predicted Profit: {result[0]}')

Multivariate Linear Regression

When dealing with multiple features, such as predicting house prices based on size and the number of bedrooms, feature scaling becomes crucial for gradient descent performance.

Feature Normalization

Features often have different scales. Normalizing them ensures that the gradient descent converges more efficiently.

def z_score_normalize(dataset):
    return (dataset - dataset.mean()) / dataset.std()

raw_data = pd.read_csv('ex1data2.txt', names=['Size', 'Bedrooms', 'Price'])
normalized_df = z_score_normalize(raw_data)

# Prepare matrices
normalized_df.insert(0, 'Ones', 1)
X_multi = normalized_df.iloc[:, :-1].values
y_multi = normalized_df.iloc[:, -1].values.reshape(-1, 1)

Evaluating Different Learning Rates

The choice of the learning rate ($̑$) significantly impacts the speed and stability of convergence.

rates = [0.001, 0.01, 0.03, 0.1]
iterations_count = 100
initial_params = np.zeros((3, 1))

plt.figure(figsize=(10, 6))
for lr in rates:
    _, costs = run_gradient_descent(X_multi, y_multi, initial_params, lr, iterations_count)
    plt.plot(costs, label=f'alpha={lr}')

plt.legend()
plt.title('Learning Rate Comparison')
plt.xlabel('Epochs')
plt.ylabel('Cost')
plt.show()

By comparing different learning rates, we can identify the threshold where the model converges quickly without overshooting the minimum.