Algorithm Overview

The AdaBoost (Adaptive Boosting) ensemble technique combines multiple weak learners to form a robust predictive model. In this implementation, the weak learner is a Decision Stump—a decision tree with a maximum depth of one. The process involves iterative training these stumps on weighted data samples, where misclassified instances receive higher emphasis in subsequent iterations.

Mathematical Foundation

• Weak Learner Weight: The contribution of a specific stump is determined by its accuracy. A stump with lower error receives a higher weight ($\alpha$):
  $\alpha_t = \frac{1}{2} \ln\left(\frac{1 - \epsilon_t}{\epsilon_t}\right)$
• Sample Weight Update: Weights for incorectly predicted samples are boosted to focus the next learner on hard-to-classsify cases.

MATLAB Implementation

1. Data Initialization

We prepare the environment by loading sample data. Here, we utilize the Fisher Iris dataset, selecting two features for binary classification simplicity.

% Initialize workspace and load data
load fisheriris
features = meas(:, 1:2); 
% Convert species to binary labels (1 for setosa, -1 for others)
labels = ones(size(species));
labels(~strcmp(species, 'setosa')) = -1;

2. Weak Learner Training Function

This function searches for the best feature and threshold (a stump) to minimize the weighted classification error.

function [bestFeat, bestThresh, bestSign] = trainStump(data, classes, sampleWeights)
    [numSamples, numFeatures] = size(data);
    minError = realmax;
    
    for fIdx = 1:numFeatures
        % Define potential split points based on feature range
        splitPoints = linspace(min(data(:,fIdx)), max(data(:,fIdx)), 20);
        
        for sIdx = 1:length(splitPoints)
            currentThresh = splitPoints(sIdx);
            
            for direction = [-1, 1]
                % Generate predictions based on threshold comparison
                predictions = (data(:,fIdx) >= currentThresh) * direction;
                
                % Calculate weighted error
                misclassified = (predictions ~= classes);
                currentError = sum(sampleWeights .* misclassified);
                
                if currentError < minError
                    minError = currentError;
                    bestFeat = fIdx;
                    bestThresh = currentThresh;
                    bestSign = direction;
                end
            end
        end
    end
end

3. Boosting Main Loop

The core loop runs for a specified number of estimators, updating weights and calculating learner importance.

function ensemble = boostedEnsemble(data, labels, numEstimators)
    [n, ~] = size(data);
    currentWeights = ones(n, 1) / n;
    ensemble.models = cell(numEstimators, 1);
    ensemble.coeffs = zeros(numEstimators, 1);
    
    for i = 1:numEstimators
        % Train a weak learner (stump)
        [feat, thresh, sign] = trainStump(data, labels, currentWeights);
        stumpPred = (data(:, feat) >= thresh) * sign;
        
        % Calculate error rate
        err = sum(currentWeights .* (stumpPred ~= labels));
        
        % Prevent division by zero or invalid log ops
        if err >= 0.5
            err = 1 - err;
            sign = -sign;
            stumpPred = (data(:, feat) >= thresh) * sign;
        end
        
        % Calculate learner coefficient (alpha)
        coeff = 0.5 * log((1 - err) / err);
        
        % Update sample weights
        currentWeights = currentWeights .* exp(-coeff * labels .* stumpPred);
        currentWeights = currentWeights / sum(currentWeights);
        
        % Store model parameters
        ensemble.models{i} = struct('f', feat, 't', thresh, 's', sign);
        ensemble.coeffs(i) = coeff;
    end
    ensemble.numEstimators = numEstimators;
end

4. Prediction Logic

Aggregates the weighted predictions from all stumps to produce the final classification.

function finalPred = classify(ensemble, testData)
    [nTest, ~] = size(testData);
    scores = zeros(nTest, 1);
    
    for i = 1:ensemble.numEstimators
        m = ensemble.models{i};
        % Prediction from single stump
        rawPred = (testData(:, m.f) >= m.t) * m.s;
        scores = scores + ensemble.coeffs(i) * rawPred;
    end
    
    % Convert aggregated scores to class labels
    finalPred = sign(scores);
end

Optimization and Tuning

Experimental Validation

We evaluate the model using a standard train-test split.

% Load Wine dataset for multiclass simulation
load wine_dataset
X = wineInputs';
Y = wineTargets;

% Split data (70/30)
cv = cvpartition(size(X,1), 'HoldOut', 0.3);
X_train = X(cv.training,:);
Y_train = Y(cv.training,:);
X_test = X(cv.test,:);
Y_test = Y(cv.test,:);

% Train the ensemble
model = boostedEnsemble(X_train, Y_train, 100);

% Test the ensemble
Y_hat = classify(model, X_test);

% Calculate performance
acc = mean(Y_hat == Y_test);
fprintf('Model Accuracy: %.2f%%\n', acc * 100);