Feature Selection Techniques in Machine Learning: Principles and Implementation

0x00 Introduction

In machine learning, data and features determine the performance ceiling, while models and algorithms merely approach this ceiling. This illustrates the critical role of feature engineering in machine learning applications. In practice, feature engineering is often the key to successful machine learning implementations.

What exactly is feature engineering?

Feature engineering is the process of leveraging domain knowledge to create features that enable machine learning algorithms to achieve optimal performance.

Feature engineering encompasses several sub-problems including Feature Selection, Feature Extraction, and Feature Construction. This article focuses on feature selection methods and their implementations.

During real-world projects, we often have access to numerous features—some carry rich information, others contain overlapping information, and some are completely irrelevant. If we use all features without filtering, we frequently encounter the curse of dimensionality, and model accuracy may even deteriorate. Therefore, we need to perform feature selection to exclude ineffective or redundant features and retain only the most valuable ones for model training.

0x01 Feature Selection Overview

1. Feature Classification by Importance

  • Relevant Features: Features that assist the learning task (such as classification) and can improve the algorithm's performance.
  • Irrelevant Features: Features that provide no benefit to our algorithm and do not enhance the algorithm's performance.
  • Redundant Features: Features that do not provide new information to our algorithm, or whose information can be derived from other features.

2. Objectives of Feature Selection

For a specific learning algorithm, determining which features are effective is initially unknown. Therefore, we must select relevant features that benefit the learning algorithm from all available features. Furthermore, the curse of dimensionality frequently arises in practical applications. Building models using only a subset of features can significantly reduce computational time and improve model interpretability.

3. Principles of Feature Selection

Obtain the smallest possible feature subset without notably reducing classification accuracy or altering the classification distribution. The feature subset should demonstrate stability and strong adaptability.

0x02 Feature Selection Methods

1. Filter Methods

Feature selection occurs first, followed by training the learner, so the selection process is independent of the learning algorithm. This approach essentially filters features upfront and then uses the filtered feature subset to train the classifier.

Core Concept: Score each feature by assigning weights that represent feature importance, then rank features based on these weights.

Common Approaches:

  • Chi-squared test
  • Information gain
  • Correlation coefficient scores

Advantages: Fast execution speed, making this a widely adopted feature selection technique.

Disadvantages: Cannot provide feedback—the feature selection criteria are determined by the search algorithm, and the learning algorithm cannot communicate feature requirements to the search algorithm. Additionally, a feature might be deemed unimportant due to arbitrary reasons during individual evaluation, but could become highly important when combined with other features.

2. Wrapper Methods

The final classifier is directly used as the evaluation function for feature selection, identifying the optimal feature subset for that specific classifier.

Core Concept: Treat subset selection as a search optimization problem. Generate various combinations, evaluate them, and compare across combinations. This transforms subset selection into an optimization problem, solvable through numerous optimization algorithms, particularly heuristic approaches such as Genetic Algorithms, Particle Swarm Optimization, Differential Evolution, and Artificial Bee Colony algorithms.

Common Approaches: Recursive Feature Elimination algorithm.

Advantages: Feature search is conducted around the learning algorithm, with selection criteria aligned with the learning algorithm's requirements. This approach can account for various learning biases of the algorithm, determining the optimal feature subset by genuinely focusing on the learning problem itself. Since the learning algorithm must execute for each tested subset, wrapper methods can capture learning biases effectively and provide substantial benefits.

Disadvantages: Significantly slower execution compared to filter methods, which explains why wrapper methods are less prevalent in practical applications.

3. Embedded Methods

Feature selection is embedded within model training. After training a model with feature selection incorporated, the resulting features and hyper-parameters can be used for subsequent training and optimization.

Core Concept: Given an established model, learn which features best improve model accuracy. During model determination, identify features that are most significant for model training.

Common Approaches: L1 regularization for feature selection (can be combined with L2 penalty for optimization), Random Forest Mean Decrease in Impurity / Mean Decrease Accuracy methods.

Advantages: Feature search is conducted around the learning algorithm and can account for various learning biases. Training occurs fewer times than wrapper methods, saving computation time.

Disadvantages: Slower execution speed.

0x03 Implementation: Low-Variance Feature Removal

This method typically serves as a preprocessing step before applying other feature selection techniques—removing features with minimal variance first, then proceeding with additional feature selection methods.

Examining the variance of a feature across samples allows us to define a threshold and eliminate features falling below that threshold.

1. Implementation Principles

  • Discrete Features: If a feature contains only values 0 and 1, with 95% of instances having a value of 1, this feature contributes minimally. If 100% of instances have the same value, the feature becomes meaningless.
  • Continuous Features: Must be discretized before applying this method.

In practice, features with over 95% identical values are uncommon, making this method somewhat limited. How ever, it serves as a valuable preprocessing step—eliminating low-variance features first, then applying more sophisticated feature selection techniques afterward.

2. Implementation Code

from sklearn.feature_selection import VarianceThreshold

data = [[1, 0, 0], [1, 0, 1], [0, 1, 0], [1, 1, 0], [1, 0, 1], [0, 1, 0]]
selector = VarianceThreshold(threshold=(0.8 * (1 - 0.8)))
transformed_data = selector.fit_transform(data)
print(transformed_data)
# Output: [[0, 1], [0, 0], [1, 0], [1, 0], [0, 0], [1, 0]]

0x04 Implementation: Univariate Feature Selection

Univariate feature selection methods evaluate each feature's relationship with the target variable independently. This approach tests every feature individually, measuring the relationship between that feature and the response variable, then discards poorly performing features. This method is straightforward, easy to implement and understand, and often provides good insights into data structure (though it may not be effective for feature optimizaton or improving generalization). Numerous improved versions and variants of this approach exist.

1. Pearson Correlation Coefficient

Pearson correlation is the simplest method for understanding relationships between features and response variables, measuring linear correlation between variables.

1) Theory

The coefficient is calculated by dividing the covariance of variables by the product of their standard deviations, **essentially a normalized covariance that eliminates dimensional influence between variables.**Covariance measures how dimensions deviate from their means—a positive covariance indicates positive correlation, while negative indicates negative correlation.

The result ranges from [-1, 1], where -1 represents complete negative correlation, +1 represents complete positive correlation, and 0 indicates no linear correlation. The absolute value represents correlation strength. Standard deviation (root mean square deviation) reflects the dispersion within a dataset.

2) Primarily applicable for continuous feature selection, not suitable for discrete features.

3) Strengths and Weaknesses

  • Advantages: Fast computation and easy calculation make correlation coefficients a common first step after data cleaning and feature extraction. Pearson correlation can characterize diverse relationships—sign indicates direction, absolute value indicates strength.
  • Disadvantages: As a feature ranking mechanism, correlation is only sensitive to linear relationships. For nonlinear relationships, even if two variables have a one-to-one correspondence, the correlation coefficient may approach zero.

4) Code Implementation

import numpy as np
from scipy.stats import pearsonr

np.random.seed(42)
sample_size = 1000
feature_a = np.random.normal(0, 1, sample_size)
print("Low noise correlation: {}".format(pearsonr(feature_a, feature_a + np.random.normal(0, 1, sample_size))))
print("High noise correlation: {}".format(pearsonr(feature_a, feature_a + np.random.normal(0, 10, sample_size))))

2. Mutual Information and Maximal Information Coefficient

When variables are not independent, we can assess their "closeness" to mutual independence by examining the Kullback-Leibler divergence between their joint probability distribution and the product of marginal distributions.

1) Mutual Information Approach

The mutual information represents the difference between entropy H(Y) and conditional entropy H(Y|X). The relationship between mutual information and conditional entropy:

This is precisely the feature selection criterion used in ID3 decision trees.

Mutual information evaluates correlations between categorical predictors and categorical targets but isn't directly convenient for feature selection:

  • It lacks normalization properties, making results incomparable across different datasets.
  • Applicable only to discrete features; continuous features require discretization before applying mutual information, and results are sensitive to the discretization method.

2) Maximal Information Coefficient Approach

Since mutual information isn't convenient for direct feature selection, the maximal information coefficient was introduced. This technique first identifies an optimal discretization method, then transforms mutual information values into a standardized metric with range [0, 1].

3) Implementation Code

x_values = np.random.normal(0, 10, 300)
transformed_z = x_values * x_values
correlation_result = pearsonr(x_values, transformed_z)
print(correlation_result[0])
# Output: -0.1

from minepy import MINE
mic_calculator = MINE()
mic_calculator.compute_score(x_values, transformed_z)
print(mic_calculator.mic())
# Output: 1.0

3. Distance Correlation

Distance correlation was developed specifically to address the limitations of Pearson correlation.

1) Theory

When Pearson correlation equals zero, this does not necessarily imply that the two variables are independent—they might have a nonlinear relationship. For example, the Pearson correlation between x and x² is 0, yet these variables are not independent.

2) Code Implementation

from scipy.spatial.distance import pdist, squareform
import numpy as np

def compute_distance_correlation(X, Y):
    """Calculate the distance correlation coefficient between two variables."""
    X = np.atleast_1d(X)
    Y = np.atleast_1d(Y)
    
    if np.prod(X.shape) == len(X):
        X = X[:, None]
    if np.prod(Y.shape) == len(Y):
        Y = Y[:, None]
    
    X = np.atleast_2d(X)
    Y = np.atleast_2d(Y)
    
    samples_count = X.shape[0]
    if Y.shape[0] != samples_count:
        raise ValueError('Sample counts must match')
    
    distance_matrix_x = squareform(pdist(X))
    distance_matrix_y = squareform(pdist(Y))
    
    mean_adj_x = distance_matrix_x - distance_matrix_x.mean(axis=0)[None, :] - distance_matrix_x.mean(axis=1)[:, None] + distance_matrix_x.mean()
    mean_adj_y = distance_matrix_y - distance_matrix_y.mean(axis=0)[None, :] - distance_matrix_y.mean(axis=1)[:, None] + distance_matrix_y.mean()
    
    cov_xy = (mean_adj_x * mean_adj_y).sum() / float(samples_count * samples_count)
    cov_xx = (mean_adj_x * mean_adj_x).sum() / float(samples_count * samples_count)
    cov_yy = (mean_adj_y * mean_adj_y).sum() / float(samples_count * samples_count)
    
    dcor = np.sqrt(cov_xy) / np.sqrt(np.sqrt(cov_xx) * np.sqrt(cov_yy))
    return dcor

4. Model-Based Feature Ranking

This approach directly uses the intended machine learning algorithm to build individual prediction models for each feature against the target variable. For nonlinear relationships, alternative approaches like tree-based methods (decision trees, random forests) or extended linear models work well. Tree-based methods are among the simplest because they handle nonlinear relationships effectively without extensive parameter tuning. The primary concern is overfitting, so tree depth should remain relatively shallow, and cross-validation is essential.

Code Implementation

from sklearn.model_selection import cross_val_score, ShuffleSplit
from sklearn.datasets import load_boston
from sklearn.ensemble import RandomForestRegressor

housing_data = load_boston()
features = housing_data["data"]
targets = housing_data["target"]
feature_names = housing_data["feature_names"]

forest_model = RandomForestRegressor(n_estimators=20, max_depth=4)
evaluation_scores = []

for feature_idx in range(features.shape[1]):
    individual_score = cross_val_score(forest_model, features[:, feature_idx:feature_idx+1], targets, 
                                        scoring="r2", cv=ShuffleSplit(len(features), 3, 0.3))
    evaluation_scores.append((round(np.mean(individual_score), 3), feature_names[feature_idx]))

print(sorted(evaluation_scores, reverse=True))

# Output: [(0.636, 'LSTAT'), (0.59, 'RM'), (0.472, 'NOX'), (0.369, 'INDUS'),
# (0.311, 'PTRATIO'), (0.24, 'TAX'), (0.24, 'CRIM'), (0.185, 'RAD'),
# (0.16, 'ZN'), (0.087, 'B'), (0.062, 'DIS'), (0.036, 'CHAS'), (0.027, 'AGE')]

5. Chi-Square Test

The chi-square test is a widely used hypothesis testing method for count data, proposed by Karl Pearson. The chi-square value describes the independence of two events or the deviation between observed and expected values. A larger chi-square value indicates greater deviation between observed and expected values, suggesting weaker independence between the two events.

1) Theory

The CHI value measures the difference between observed and theoretical values. Dividing by the expected value T normalizes for variations caused by different expectations.

  • Absolute magnitude of deviation between observed and theoretical values (amplified by squaring)
  • Relative magnitude of deviation compared to theoretical values

2) Implementation Process

Larger CHI values indicate that two variables are unlikely to be independent—the higher the CHI value, the stronger the correlation between variables.

a. For feature variables x₁, x₂, …, xₙ and classification variable y, calculate CHI(x₁,y), CHI(x₂,y), …, CHI(xₙ,y), then rank features in descending order by CHI values.

b. Choose an appropriate threshold—features exceeding the threshold are retained, those below are eliminated. The resulting feature subset becomes the input for model training.

3) Applicable only to discrete features in classification problems; not suitable for continuous features in classification or regression problems.

4) Code Implementation

Practical implementations include:

  • sklearn.feature_selection.SelectKBest: Returns the top k best features
  • sklearn.feature_selection.SelectPercentile: Returns the top r% performing features
from sklearn.feature_selection import SelectKBest, chi2

# Select the top 5 features with highest correlation to target
top_features = SelectKBest(chi2, k=5).fit_transform(feature_matrix, target_labels)
print(top_features.shape)
# Output: (27, 5)

0xFF Summary

  1. The low-variance feature removal method serves as preprocessing before feature selection—eliminating low-variance features first, then applying other selection methods. When computational resources are abundant and preserving all information is desired, set a higher threshold or filter only features with single unique values.
  2. Univariate feature selection helps understand data structure and characteristics, and can exclude irrelevant features, but cannot identify redundant features.

Both low-variance removal and univariate feature selection fall under filter-based methods, where the learning algorithm cannot communicate feature requirements to the search algorithm. To genuinely focus on the learning problem itself, the following article will continue with wrapper methods and embedded method principles and implementations.

References:

Tags: machine-learning feature-engineering Feature-Selection filter-methods pearson-correlation

Posted on Sat, 25 Jul 2026 16:18:57 +0000 by detrox

Hot Tags

©2026 Freaks City