A Beginner‘s Guide to Low Variance Filters for Feature Selection

Introduction

In machine learning, more data isn‘t always better. While additional features can sometimes improve model performance, they can also make training more computationally expensive and increase the risk of overfitting. Feature selection is the process of identifying the most relevant features and ignoring the rest. It‘s a crucial step in the machine learning pipeline that can lead to more efficient and generalizable models.

One simple yet effective feature selection technique is the low variance filter. In this guide, we‘ll dive deep into what low variance filters are, why they‘re useful, and how to implement them in Python.

The Curse of Dimensionality

Before we jump into low variance filters, it‘s important to understand the motivation behind feature selection. In machine learning, the curse of dimensionality refers to the phenomenon where increasing the number of features can actually degrade model performance if not done carefully. This happens for a few reasons:

1. Increased computational complexity and training time
2. Higher risk of finding spurious correlations
3. Increased difficulty of data visualization and model interpretation
4. Need for more training data to produce a generalizable model

The following chart shows how increasing the number of features increases the amount of data needed to maintain the same level of model performance:

As the number of features grows, the amount of data needed to reach a given performance level grows exponentially. This is why feature selection is so important – it helps us tame the curse of dimensionality and build models that are both accurate and computationally tractable.

What is Feature Variance?

Variance is a statistical measure of how much a feature‘s values are spread out from their mean. It‘s calculated as the average squared deviation from the mean. The formula for population variance is:

$\sigma^2 = \frac{\sum_{i=1}^{N} (x_i – \mu)^2}{N}$

Where:

• $\sigma^2$ is variance
• $\mu$ is the mean of the feature values
• $N$ is the number of instances
• $x_i$ is the feature value for instance $i$

A feature with high variance has values that are widely dispersed from the mean, while a low variance feature has values tightly clustered around the mean. Here‘s a visualization contrasting a low variance feature with a high variance feature:

Intuitively, we can see that the high variance feature contains more information because it has a wider range of values. The low variance feature, in contrast, has almost no distinguishing information because most instances have the same value. This is the key insight behind low variance filters – features with extremely low variance are unlikely to be informative for predicting the target variable.

Implementing a Low Variance Filter

Now that we understand what feature variance is and why it matters, let‘s see how to implement a low variance filter in Python. We‘ll use scikit-learn‘s VarianceThreshold class which removes all features whose variance is below a specified threshold.

First, we load the data and separate the features from the target variable:

import pandas as pd
from sklearn.feature_selection import VarianceThreshold

data = pd.read_csv("dataset.csv")
X = data.drop("target", axis=1)  
y = data["target"]

Next, we create a VarianceThreshold object and specify the variance threshold. The threshold is the minimum variance a feature must have to be retained. It‘s expressed as a fraction of the maximum variance of any feature. For example, a threshold of 0.1 means that a feature must have at least 10% of the variance of the highest variance feature to be kept.

selector = VarianceThreshold(threshold=0.1)

We then fit the selector to the data and transform the features to remove the low variance columns:

X_selected = selector.fit_transform(X)

We can see how many features were removed:

print(f"Original feature count: {X.shape[1]}")
print(f"Selected feature count: {X_selected.shape[1]}")

Original feature count: 50
Selected feature count: 32

In this case, the low variance filter removed 18 features that fell below the 10% max variance threshold.

Tuning the Variance Threshold

Choosing the right variance threshold is crucial to getting the most benefit from a low variance filter. Set the threshold too low and you‘ll keep noisy, uninformative features. Set it too high and you risk discarding relevant signals.

As a general workflow, it‘s a good idea to start with a low threshold (1-5% of max variance) and gradually increase it while monitoring model performance. The following code shows how you could implement this threshold tuning procedure:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

thresholds = np.linspace(0.01, 0.5, 20)

scores = []
for t in thresholds:
    selector = VarianceThreshold(threshold=t)
    X_train_selected = selector.fit_transform(X_train)
    X_test_selected = selector.transform(X_test)

    model = LogisticRegression()
    model.fit(X_train_selected, y_train)
    y_pred = model.predict(X_test_selected)

    accuracy = accuracy_score(y_test, y_pred)
    scores.append(accuracy)

best_threshold = thresholds[np.argmax(scores)]  
print(f"Best threshold: {best_threshold:.2f}")
print(f"Best accuracy: {np.max(scores):.2f}")

This code tries 20 thresholds evenly spaced between 1% and 50% of maximum variance. For each threshold, it fits a logistic regression model and evaluates accuracy on a held-out test set. Finally, it reports the threshold that led to the highest test accuracy.

Here‘s an example of what the threshold vs accuracy plot might look like:

In this case, the optimal variance threshold was around 0.15 which resulted in a model accuracy of approximately 89%.

Your ideal threshold will depend on your specific dataset and modeling goals, but this example illustrates the general process of tuning the variance threshold by empirical experimentation.

Handling Categorical Features

So far, we‘ve focused on applying low variance filters to numerical features. However, the same concept can be extended to categorical features by looking at the frequency distribution of categories rather than variance.

With categorical features, we want to identify features where a single category makes up the vast majority of the values. For example, imagine a "country" feature where 95% of the values are "USA". This feature would be a candidate for removal because the dominant category doesn‘t provide much distinguishing information.

To identify low variance categorical features, we can use the DictVectorizer from scikit-learn to convert categorical variables to a one-hot encoded representation and then apply VarianceThreshold as before.

from sklearn.feature_extraction import DictVectorizer

# Convert dataframe to dict records
records = X.to_dict(orient="records")

# One-hot encode categorical features
vectorizer = DictVectorizer(sparse=False)
X_encoded = vectorizer.fit_transform(records)

# Apply variance threshold
selector = VarianceThreshold(threshold=0.01)
X_selected = selector.fit_transform(X_encoded)

In this case, we set the variance threshold to 1% of the maximum variance. This means that for a binary categorical feature to be retained, each category must appear at least 1% of the time.

Integrating Low Variance Filters in a ML Pipeline

Low variance filters are typically used as a preprocessing step before training a machine learning model. Here‘s an example of what a complete pipeline might look like:

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier

pipeline = Pipeline([
    ("imputer", SimpleImputer()), 
    ("scaler", StandardScaler()),
    ("variance_filter", VarianceThreshold(threshold=0.1)),
    ("classifier", RandomForestClassifier())
])

pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)

This pipeline first imputes missing values, then standardizes the features to have zero mean and unit variance. Next, it applies a low variance filter with a 10% max variance threshold. Finally, it trains a random forest classifier on the selected features.

The nice thing about scikit-learn pipelines is that they ensure the same transformations are applied to the train and test data and they make it easy to experiment with different models and preprocessing steps.

Advanced Topics and Considerations

While we‘ve covered the core concepts behind low variance filters, there are a few advanced topics worth mentioning.

One issue to be aware of is the potential presence of near-zero variance predictors. These are features that have only a single unique value (i.e. zero variance) except for a very small number of instances.

For example, imagine a feature that has the value "A" for 99.9% of instances and "B" for the remaining 0.1%. This feature would be almost perfectly predictive of the target variable in the training set but is unlikely to generalize. A simple variance threshold would retain this feature.

To handle near-zero variance predictors, a common approach is to combine a variance threshold with a frequency threshold for the dominant category (e.g. discard features where a single category makes up more than 99.9% of the values).

Another consideration is that variance is sensitive to the scale of the features. Therefore, it‘s a good practice to standardize or normalize the data before applying a variance filter, as we saw in the pipeline example.

Finally, there are some cases where low variance features can still be predictive when combined with other features. A variance filter considers each feature in isolation and cannot detect interactions. Therefore, it‘s a good idea to combine low variance filters with other feature selection techniques to get a more comprehensive view of the feature space.

Conclusion

We‘ve covered a lot of ground in this guide to low variance filters. The key points to remember are:

• Low variance filters are a simple and computationally efficient way to remove non-informative features
• They work by discarding features whose variance falls below a specified threshold
• The variance threshold can be tuned based on model performance and domain knowledge
• Low variance filters can be applied to both numerical and categorical data
• It‘s important to standardize features and watch out for near-zero variance predictors
• Low variance filters are a useful preprocessing step but should be combined with other feature selection techniques

If you‘re new to feature selection, I encourage you to start by applying a low variance filter to your own datasets and seeing how it impacts your model results. Play around with different thresholds and see how the feature set changes.

While low variance filters are not a panacea, they are a valuable tool to have in your machine learning toolbox. By removing noisy and irrelevant features, they can help you build models that are more accurate, interpretable, and computationally efficient.

References

• Kuhn, M., & Johnson, K. (2013). Applied predictive modeling. Springer Science & Business Media.
• Zheng, A., & Casari, A. (2018). Feature engineering for machine learning: principles and techniques for data scientists. O‘Reilly Media, Inc.
• Cai, J., Luo, J., Wang, S., & Yang, S. (2018). Feature selection in machine learning: A new perspective. Neurocomputing, 300, 70-79.
• Bolón-Canedo, V., Sánchez-Maroño, N., & Alonso-Betanzos, A. (2015). A review of feature selection methods on synthetic data. Knowledge and information systems, 34(3), 483-519.