# 10 Techniques for Dealing with Class Imbalance in Machine Learning

- Canonical: https://33rdsquare.com/10-techniques-to-deal-with-class-imbalance-in-machine-learning/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

Class imbalance is a common problem in real-world machine learning applications. It occurs when one class (the "majority" class) has significantly more examples than the other class(es) (the "minority" class). This situation arises frequently in contexts like fraud detection, medical diagnosis, and fault monitoring systems.

Most standard machine learning algorithms expect balanced class distributions. When presented with imbalanced datasets, these algorithms tend to favor the majority class, resulting in poor predictive performance for the minority class. However, the minority class is often the class of interest from an application perspective.

In this post, we‘ll explore 10 techniques you can use to handle imbalanced datasets and build classifiers with good performance on both the majority and minority classes. We‘ll cover methods ranging from simple resampling to more advanced algorithmic approaches, with code examples showing how to implement them in Python.

## The Accuracy Paradox

Before diving into specific techniques, it‘s important to highlight the limitations of using accuracy to evaluate classifiers trained on imbalanced data.

Suppose we have a dataset where 95% of examples are negative and 5% are positive. A model that always predicts the negative class would achieve 95% accuracy, but it would never detect any positive cases. Clearly accuracy is misleading us about the model‘s true performance.

Alternative metrics like F1-score, ROC AUC, and precision-recall curves are more informative when dealing with class imbalance. These metrics can reveal poor performance on the minority class that accuracy obscures.

## 1. Undersampling

One straightforward way to address class imbalance is to discard examples from the majority class until the classes are balanced. This is called undersampling.

The main advantage of undersampling is simplicity. It‘s easy to implement and doesn‘t introduce any new examples into the dataset.

However, undersampling has some drawbacks. By discarding potentially useful data, the model may miss out on important patterns. There‘s also a risk of discarding too much data, leaving too few examples to train an effective model.

Here‘s how to perform random undersampling using the imbalanced-learn library in Python:

```
from imblearn.under_sampling import RandomUnderSampler

undersample = RandomUnderSampler(sampling_strategy=‘majority‘)
X_under, y_under = undersample.fit_resample(X, y)
```

## 2. Oversampling

Oversampling takes the opposite approach of undersampling. Instead of removing examples from the majority class, oversampling duplicates examples from the minority class.

The advantage of oversampling is that no data is discarded. The model is trained on all available examples.

However, oversampling can lead to overfitting, since the same minority examples may be seen multiple times during training. There is also a risk of exacerbating class overlap if minority examples are replicated in regions where the classes are already mixed.

Imbalanced-learn makes oversampling simple:

```
from imblearn.over_sampling import RandomOverSampler

oversample = RandomOverSampler(sampling_strategy=‘minority‘)
X_over, y_over = oversample.fit_resample(X, y)
```

## 3. SMOTE

SMOTE (Synthetic Minority Oversampling TEchnique) is a more sophisticated oversampling approach. Instead of simply duplicating minority examples, SMOTE generates new synthetic examples.

For each minority example, SMOTE finds its k-nearest minority neighbors. It then creates new examples by interpolating between the example and its neighbors.

SMOTE can be more effective than simple oversampling because the synthetic examples it generates spread the minority class decision boundary into the majority class space. However, by creating artificial examples, SMOTE risks introducing noisy examples if interpolation occurs between examples of different classes.

Using SMOTE with imbalanced-learn is straightforward:

```
from imblearn.over_sampling import SMOTE

smote = SMOTE(sampling_strategy=‘minority‘, k_neighbors=5)
X_smote, y_smote = smote.fit_resample(X, y)
```

## 4. Class Weights

Many machine learning algorithms allow you to assign different misclassification costs to different classes through a hyperparameter called class weights.

Increasing the cost of misclassifying minority examples forces the algorithm to focus more on the minority class. In effect, this makes misclassifying a minority example more expensive than misclassifying a majority example.

The advantage of class weights is that they don‘t involve resampling the dataset. You use the original data, but alter the algorithm‘s optimization function.

The main disadvantage is that finding the optimal weights can require extensive hyperparameter tuning. Overly high weights can also cause the model to overfit the minority class.

Here‘s an example of using class weights with scikit-learn‘s random forest classifier:

```
from sklearn.ensemble import RandomForestClassifier

weights = {0: 1.0, 1: 10.0} # 10x higher weight for minority class
rf = RandomForestClassifier(class_weight=weights)
rf.fit(X_train, y_train)
```

## 5. Balanced Ensemble Methods

Ensemble methods like bagging and boosting can be adapted to handle class imbalance. The basic idea is to build each individual model in the ensemble using a balanced subsample of the data.

For example, the balanced random forest randomly undersamples the majority class when bootstrapping data for each tree. By training each tree on a balanced subsample, the ensemble as a whole better captures the minority class.

The main advantage of balanced ensembles is that they don‘t discard majority examples like undersampling does. All the data is used, but each model only sees a balanced portion of it.

The disadvantage is potentially longer training times, since multiple balanced subsamples are used rather than the entire dataset at once.

Imbalanced-learn offers several balanced ensemble methods:

```
from imblearn.ensemble import BalancedRandomForestClassifier

brf = BalancedRandomForestClassifier(sampling_strategy=‘auto‘)
brf.fit(X_train, y_train)
```

## 6. Anomaly Detection

Taking a different perspective, we can view the minority class as outliers or anomalies against a background of majority class examples.

Anomaly detection algorithms like isolation forest, one-class SVM, and local outlier factor don‘t require balanced classes. They learn the concept of "normal" from the majority class, then identify the minority class as anomalies.

The advantage of anomaly detection is that it‘s specifically designed for imbalanced data. It doesn‘t need to see many minority examples to identify them.

However, anomaly detection may not be suitable if the minority class has a lot of inherent variation. The algorithms assume anomalies are rare and different from normal; they may fail to detect minority examples that blend in with the majority.

Here‘s an example using scikit-learn‘s isolation forest:

```
from sklearn.ensemble import IsolationForest

isof = IsolationForest(contamination=0.05) # expect 5% minority class
isof.fit(X_train)

outlier_scores = isof.decision_function(X_test)
outlier_predictions = isof.predict(X_test)
```

## 7. Combining Resampling with Algorithmic Approaches

The techniques we‘ve covered so far broadly fall into two categories:

1. Resampling methods that modify the training data to be more balanced
2. Algorithmic methods that adapt the learning algorithm itself to account for class imbalance

These approaches aren‘t mutually exclusive. In fact, combining resampling with algorithmic techniques often achieves the best results.

For example, you could oversample the minority class using SMOTE, then train a random forest with class weights on the resampled data. The oversampling addresses the fundamental imbalance, while the class weights fine-tune the algorithm‘s focus on the minority class.

## 8. Threshold Moving

Many classifiers produce probability estimates or decision function values that are thresholded to make final predictions. With imbalanced classes, the default threshold may not be optimal.

Threshold moving involves adjusting the decision threshold to account for the imbalanced class priors. Intuitively, if the model is more likely to see majority examples, its threshold for predicting the minority class should be lower.

The optimal threshold can be selected by plotting precision-recall curves or ROC curves for different threshold values and finding the best trade-off point.

Here‘s an example of threshold tuning using scikit-learn:

```
from sklearn.metrics import precision_recall_curve

probs = clf.predict_proba(X_test)[:, 1] # positive class probabilities
precision, recall, thresholds = precision_recall_curve(y_test, probs)

# select threshold with desired precision-recall balance
chosen_threshold = thresholds[np.argmax(precision + recall)]

y_pred = (probs >= chosen_threshold).astype(int)
```

## 9. Resource-Constrained Oversampling

In some applications, false negatives (misclassifying minority examples) are much more costly than false positives. For example, in cancer diagnosis, mistakenly declaring a malignant tumor benign is far worse than flagging a benign tumor for further testing.

With limited resources for handling false positives, we can perform oversampling to a point that matches the resource constraints. The goal is to catch as many true positives as possible while generating only as many false alarms as can be afforded.

```
from imblearn.over_sampling import SMOTE

max_false_positives = 100 # only have resources for 100 false alarms

for ratio in np.arange(0.1, 1.0, 0.1): # try different oversampling ratios
    smote = SMOTE(sampling_strategy=ratio)
    X_res, y_res = smote.fit_resample(X_train, y_train)
    clf.fit(X_res, y_res)

    false_positives = (clf.predict(X_valid) != y_valid).sum()
    if false_positives <= max_false_positives:
        break # stop when false positives exceed budget
```

## 10. Stratified K-Fold Validation

When evaluating models on imbalanced datasets, it‘s important to use stratified sampling for cross-validation. Stratified sampling ensures each fold has the same class balance as the original dataset.

Without stratification, some folds may end up with very few or even zero minority examples, leading to unreliable performance estimates.

Scikit-learn‘s `StratifiedKFold` makes stratified cross-validation easy:

```
from sklearn.model_selection import StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True)

for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    clf.fit(X_train, y_train)
    scores.append(clf.score(X_test, y_test))
```

## Conclusion

Class imbalance is a frequent issue in real-world machine learning that can lead to misleading results if not handled appropriately. Accuracy alone is not a reliable metric for evaluating classifiers on imbalanced data.

Fortunately, there are many techniques available for dealing with class imbalance, including resampling methods, algorithmic approaches, and hybrid strategies. The best approach depends on the specific characteristics of the data and the goals of the application.

When working with imbalanced datasets, it‘s crucial to use appropriate evaluation metrics like F1-score, ROC AUC, and precision-recall curves. These metrics provide a more complete picture of performance than accuracy.

Ultimately, the key to handling class imbalance effectively is being aware of its presence and its potential impacts. By thoughtfully applying the techniques covered in this post, you can build robust models that perform well on both majority and minority classes.

---

Source: [10 Techniques for Dealing with Class Imbalance in Machine Learning](https://33rdsquare.com/10-techniques-to-deal-with-class-imbalance-in-machine-learning/)