In this tutorial, we'll explore the fundamental concepts of linear and polynomial regression, two essential techniques in machine learning for modeling relationships between variables. We'll examine their mathematical foundations and practical implementations using Python's scikit-learn library.

Regression Analysis Overview

Regression analysis helps us understand how the dependent variable changes when one or more independent variables are varied. Linear regression models relationships using straight lines, while polynomial regression allows for more complex, curved relationships between variables.

Basic Linear Regression

Linear regression aims to find the best-fitting straight line through the data points. The "perfect line" minimizes the distance between all data points and the line itself. This is typically achieved using the least squares method.

The mathematical formula for calculating this error is:

$ sum_{i=1}^{n} (y_i - f(x_i))^2$

We seek to find parameters that minimize the total error (distance) between sample points and the fitted line. The best-fit line can be represented by:

Y = a + bX

Where X is the explanatory variable, Y is the dependent variable, b is the slope, and a is the y-intercept (the value of Y when X = 0).

A good linear regression model will have a high correlation coefficient (closer to 1) obtained through least squares regression. The correlation coefficient (Pearson correlation) indicates:

• The strength and direction of linear relationships between variables
• Not the slope of the relationship
• Not适用于 nonlinear relationships

Data Preparation

Let's begin by loading and preparing our dataset. We'll use the US pumpkins dataset to demonstrate regression techniques.

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from datetime import datetime
from sklearn.preprocessing import LabelEncoder

# Load the dataset
pumpkin_data = pd.read_csv('../data/US-pumpkins.csv')

# Filter for bushel packages and select relevant columns
pumpkin_data = pumpkin_data[pumpkin_data['Package'].str.contains('bushel', case=True, regex=True)]
selected_columns = ['Package', 'Variety', 'City Name', 'Low Price', 'High Price', 'Date']
pumpkin_data = pumpkin_data.loc[:, selected_columns]

# Calculate average price
average_price = (pumpkin_data['Low Price'] + pumpkin_data['High Price']) / 2

# Extract month and day of year from date
month_data = pd.DatetimeIndex(pumpkin_data['Date']).month
day_of_year_data = pd.to_datetime(pumpkin_data['Date']).apply(lambda dt: (dt-datetime(dt.year,1,1)).days)

# Create new dataframe with processed data
processed_pumpkins = pd.DataFrame({
    'Month': month_data, 
    'DayOfYear' : day_of_year_data, 
    'Variety': pumpkin_data['Variety'], 
    'City': pumpkin_data['City Name'], 
    'Package': pumpkin_data['Package'], 
    'Low Price': pumpkin_data['Low Price'],
    'High Price': pumpkin_data['High Price'], 
    'Price': average_price
})

# Adjust prices based on package size
processed_pumpkins.loc[processed_pumpkins['Package'].str.contains('1 1/9'), 'Price'] = average_price/1.1
processed_pumpkins.loc[processed_pumpkins['Package'].str.contains('1/2'), 'Price'] = average_price*2

# Convert categorical variables to numerical
processed_pumpkins.iloc[:, 0:-1] = processed_pumpkins.iloc[:, 0:-1].apply(LabelEncoder().fit_transform)

processed_pumpkins.head()

Visualizing Relationships

Let's visualize the relationship between city and price to understand potential correlations:

plt.scatter('City', 'Price', data=processed_pumpkins)
plt.xlabel('City')
plt.ylabel('Price')
plt.title('Price Distribution by City')
plt.show()

The city field was converted to numerical values using LabelEncoder to enable mathematical operations. While visual inspection can reveal patterns, we need quantitative measures to assess relationships accurately.

city_price_correlation = processed_pumpkins['City'].corr(processed_pumpkins['Price'])
print(f"City-Price correlation: {city_price_correlation:.4f}")
# Output: 0.3236

The correlation of approximately 0.32 indicates a weak relationship between city and price. Let's examine other potential predictors:

package_price_correlation = processed_pumpkins['Package'].corr(processed_pumpkins['Price'])
print(f"Package-Price correlation: {package_price_correlation:.4f}")
# Output: 0.6062

The package type shows a stronger correlation with price (0.61). To systematically explore relationships, we can create a correlation heatmap:

import seaborn as sns

# Select relevant columns for correlation analysis
analysis_columns = ['Package', 'Variety', 'City', 'Month', 'Price']
correlation_data = processed_pumpkins[analysis_columns]

# Calculate correlation matrix
correlation_matrix = correlation_data.corr()

# Create heatmap
plt.figure(figsize=(10, 8))
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', vmin=-1, vmax=1)
plt.title('Correlation Heatmap')
plt.show()

Building Linear Regression Model

Based on our correlation analysis, package type shows the strongest relationship with price. Let's build a linear regression model using this feature:

from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error
from sklearn.model_selection import train_test_split

# Prepare data for modeling
model_data = processed_pumpkins[['Package', 'Price']]
X = model_data.values[:, :1]
y = model_data.values[:, 1:2]

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize and train the linear regression model
linear_model = LinearRegression()
linear_model.fit(X_train, y_train)

# Make predictions
predictions = linear_model.predict(X_test)

# Evaluate model performance
train_score = linear_model.score(X_train, y_train)
print(f"Model Accuracy: {train_score:.4f}")
# Output: 0.3315

The model accuracy of 0.33 indicates that while there's some relationship, it's not very strong. Let's visualize the model:

plt.scatter(X_test, y_test, color='black', label='Actual')
plt.plot(X_test, predictions, color='blue', linewidth=2, label='Predicted')
plt.xlabel('Package Type')
plt.ylabel('Price')
plt.title('Linear Regression: Package vs Price')
plt.legend()
plt.show()

Polynomial Regression

Linear relationships don't always capture the complexity of real-world data. Polynomial regression allows us to model nonlinear relationships by transforming features into polynomial terms.

# Select features for polynomial regression
poly_features = ['Variety', 'Package', 'City', 'Month', 'Price']
poly_data = processed_pumpkins[poly_features]

# Prepare data
X_poly = poly_data.iloc[:, 3:4].values  # Using Month as feature
y_poly = poly_data.iloc[:, 4:5].values  # Price as target

# Split data
X_train_poly, X_test_poly, y_train_poly, y_test_poly = train_test_split(X_poly, y_poly, test_size=0.2, random_state=42)

# Create polynomial regression pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

poly_pipeline = make_pipeline(PolynomialFeatures(4), LinearRegression())
poly_pipeline.fit(X_train_poly, y_train_poly)
poly_predictions = poly_pipeline.predict(X_test_poly)

Understanding PolynomialFeatures

PolynomialFeatures transforms input features into polynomial combinations, enabling the model to capture nonlinear relationships:

# Demonstrate PolynomialFeatures transformation
sample_data = np.array([1, 2, 3, 4, 5]).reshape(-1, 1)
print("Original data:")
print(sample_data)

poly_transformer = PolynomialFeatures(4)
transformed_data = poly_transformer.fit_transform(sample_data)
print("\nTransformed data:")
print(transformed_data)

Output:

Original data:
[[1]
 [2]
 [3]
 [4]
 [5]]

Transformed data:
[[  1.   1.   1.   1.   1.]
 [  1.   2.   4.   8.  16.]
 [  1.   3.   9.  27.  81.]
 [  1.   4.  16.  64. 256.]
 [  1.   5.  25. 125. 625.]]

This transformation allows linear regression to model nonlinear relationships by considering polynomial combinations of features.

Visualizing Polynomial Regression

# Create visualization of polynomial regression
df_poly = pd.DataFrame({'x': X_test_poly[:,0], 'y': poly_predictions[:,0]})
df_poly.sort_values(by='x', inplace=True)
sorted_points = df_poly.to_numpy()

plt.figure(figsize=(10, 6))
plt.scatter(X_poly, y_poly, color='black', label='Actual Data')
plt.plot(sorted_points[:, 0], sorted_points[:, 1], color='blue', linewidth=2, label='Polynomial Fit')
plt.xlabel('Month')
plt.ylabel('Price')
plt.title('Polynomial Regression: Month vs Price')
plt.legend()
plt.show()

Making Predictions

# Predict price for a specific month
month_prediction = poly_pipeline.predict(np.array([[2.75]]))
print(f"Predicted price for month 2.75: ${month_prediction[0][0]:.2f}")
# Output: 46.35

This demonstrates how polynomial regression can capture more complex patterns in the data compared to simple linear regression.