5 Techniques to Handle Imbalanced Data for Classification Problems
Introduction
Imbalanced data is a common problem in real-world classification tasks. It occurs when one class (the "majority" class) has significantly more examples than the other class or classes (the "minority" classes).
For example, suppose you‘re building a model to detect fraudulent credit card transactions. In a typical dataset, the vast majority of transactions (99.9%) will be non-fraudulent, while only a tiny fraction (0.1%) will be fraudulent. This is an imbalanced dataset with a 999:1 class ratio.
Most standard classification algorithms expect balanced class distributions and equal misclassification costs. When presented with imbalanced data, these algorithms fail to properly represent the distributive characteristics of the data. They tend to bias towards the majority class and have poor predictive accuracy for the minority class.
This is problematic because in most imbalanced domains, the minority class is of greater interest and importance than the majority class – it‘s more important to detect the rare fraudulent transactions than the non-fraudulent ones.
Fortunately, there are a number of techniques we can use to handle imbalanced datasets and build better performing, more useful classification models. In this post, we‘ll explore five of the most popular and effective methods:
- Resampling the dataset
- Synthesizing new minority class instances
- Adjusting class weights
- Using appropriate evaluation metrics
- Using special-purpose learning algorithms
We‘ll explain each technique, discuss its pros and cons, and show how to implement it in Python with worked code examples. By the end, you‘ll be well equipped to tackle imbalanced classification problems and build high-performing models that work well on real-world data.
Let‘s dive in!
1. Resampling the Dataset
One of the simplest and most popular methods for dealing with imbalanced data is to resample the training set. The idea is to modify the class distribution of the training data so that the model sees a more balanced class distribution during training.
There are two main approaches to resampling:
- Oversampling: Increasing the number of minority class examples
- Undersampling: Reducing the number of majority class examples
Oversampling
In oversampling, we duplicate examples from the minority class to increase its representation in the training set. One simple approach is to randomly replicate minority class examples until the class distribution is more balanced.
For example, if we have 100 majority class examples and only 10 minority class examples, we could randomly duplicate the minority examples 9 more times each to get a total of 100 minority examples.
The benefit of oversampling is that no information from the original training set is lost, since all examples are kept. The downside is that it increases the training set size, which increases memory usage and training time. It can also lead to overfitting if examples are duplicated too many times.
Here‘s how we can implement random oversampling in Python using the imbalanced-learn library:
from imblearn.over_sampling import RandomOverSampler
oversample = RandomOverSampler(sampling_strategy=‘minority‘)
X_over, y_over = oversample.fit_resample(X, y)Undersampling
In undersampling, we remove examples from the majority class to reduce its representation in the training set. Again, the simplest approach is random undersampling, where we randomly drop majority class examples until the class distribution is more balanced.
For example, if we have 1000 majority class examples and 100 minority class examples, we could randomly remove 900 majority examples to get a balanced 100:100 class ratio.
The benefit of undersampling is that it reduces the training set size, which lowers memory usage and speeds up training. It can also help the model to generalize better by preventing it from biasing too strongly towards the majority class.
The downside is that by removing examples, we lose potentially useful information that the model could have learned from. It can also lead to underfitting if too many useful majority examples are removed.
Here‘s how to implement random undersampling in Python with imbalanced-learn:
from imblearn.under_sampling import RandomUnderSampler
undersample = RandomUnderSampler(sampling_strategy=‘majority‘)
X_under, y_under = undersample.fit_resample(X, y)Hybrid Approach
A third option is to combine oversampling and undersampling into a hybrid approach. One effective method is to oversample the minority class with replacement, and undersample the majority class without replacement.
This preserves useful information in the minority class through duplication, while reducing the skew of the majority class. It keeps the training set size the same as the original.
from imblearn.combine import SMOTETomek
resample = SMOTETomek(sampling_strategy=‘auto‘)
X_resampled, y_resampled = resample.fit_resample(X, y)Overall, resampling is a good first approach for imbalanced classification. It‘s simple to implement and often quite effective. The best resampling strategy will depend on your particular dataset and problem.
2. Synthesizing New Minority Class Examples
Simply duplicating minority class examples, as in random oversampling, doesn‘t add any new information to the model. More sophisticated oversampling techniques attempt to synthesize completely new minority class examples based on the examples that already exist.
The most popular of these is SMOTE (Synthetic Minority Over-sampling Technique). SMOTE looks at the feature space for each target class and considers its nearest neighbors. It then generates new examples that combine features of the target case with features of its neighbors. This approach increases the features available for each class and makes the samples more general.
For example, if we have a minority class example with feature vector [1, 3, 5] and one of its nearest neighbors is [2, 1, 6], SMOTE might synthesize a new example with feature vector [1.5, 2, 5.5], which is halfway between the two examples in feature space.
Here‘s how to implement SMOTE in Python:
from imblearn.over_sampling import SMOTE
smote = SMOTE(sampling_strategy=‘auto‘)
X_smote, y_smote = smote.fit_resample(X, y) A variation of SMOTE is ADASYN (Adaptive Synthetic), which generates different numbers of synthetic examples for each minority example, with more synthetic examples created for minority examples that are harder to learn.
Synthesizing new examples avoids the overfitting risk of random oversampling, but it can be more computationally expensive. SMOTE and ADASYN often outperform plain oversampling and are good next steps if resampling alone isn‘t sufficient.
3. Adjusting Class Weights
Another approach to imbalanced classification is to adjust the "weight" of each class during training. Intuitively, we want to penalize the model more for misclassifying minority class examples, since those are the examples we care most about getting right.
In practice, this means setting a higher misclassification cost for the minority class than the majority class. Most classification algorithms in Python have a class_weight parameter that lets you do exactly this.
For example, if we have a 1:100 minority:majority class ratio, we might assign a weight of 1 to the majority class and a weight of 100 to the minority class, to balance out their impact on the model‘s loss function.
Here‘s how to set class weights in Python with scikit-learn:
from sklearn.ensemble import RandomForestClassifier
# Compute class weights that inversely proportional to class frequencies
weights = {0: 1.0, 1: y.shape[0] / y.sum()}
rf = RandomForestClassifier(class_weight=weights)
rf.fit(X_train, y_train)
rf.predict(X_test)Adjusting class weights doesn‘t require any data resampling, so it‘s computationally cheaper than oversampling and doesn‘t risk losing information like undersampling. It effectively makes the model pay more attention to the minority class without actually changing the class distribution.
Class weights are supported by many algorithms, including decision trees, random forests, SVMs, and logistic regression. It‘s a good option when you want to stick with standard algorithms and avoid resampling.
4. Using Appropriate Evaluation Metrics
With imbalanced classes, standard accuracy is not an appropriate metric for assessing model performance. For example, a model that always predicts the majority class will have high accuracy, but it will completely fail at identifying the important minority class.
For imbalanced problems, metrics that take class imbalance into account are more appropriate, such as:
- Confusion matrix: Shows the number of true positives, true negatives, false positives, and false negatives
- Precision: What fraction of positive predictions were actually correct? Precision = TP / (TP + FP)
- Recall (aka sensitivity or true positive rate): What fraction of actual positive examples were correctly identified? Recall = TP / (TP + FN)
- F1 score: Harmonic mean of precision and recall, a good overall metric. F1 = 2 (precision recall) / (precision + recall)
- ROC curves: Plots true positive rate against false positive rate at different classification thresholds
Here‘s how to compute these metrics in Python with scikit-learn:
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, roc_auc_score
y_pred = model.predict(X_test)
print(confusion_matrix(y_test, y_pred))
print("Precision:", precision_score(y_test, y_pred))
print("Recall:", recall_score(y_test, y_pred))
print("F1 score:", f1_score(y_test, y_pred))
print("ROC AUC:", roc_auc_score(y_test, model.predict_proba(X_test)[:,1]))When evaluating imbalanced classifiers, it‘s important to look at multiple metrics to get a complete picture of performance. Plotting precision-recall curves or ROC curves can also help with selecting an appropriate classification threshold.
5. Using Special-Purpose Learning Algorithms
Finally, certain learning algorithms are designed specifically to handle imbalanced data. These methods typically combine data resampling techniques with standard learning algorithms into ensemble models.
A few examples:
- Balanced Random Forest: Undersamples the majority class for each bootstrap sample when building the ensemble of decision trees.
- EasyEnsemble: Bags multiple AdaBoost learners trained on different balanced bootstrap samples.
- RUSBoost: Combines random undersampling with the standard boosting procedure.
Here‘s an example using BalancedRandomForest in Python:
from imblearn.ensemble import BalancedRandomForestClassifier
brf = BalancedRandomForestClassifier(n_estimators=100)
brf.fit(X_train, y_train)
brf.predict(X_test)Special-purpose algorithms can achieve better performance on imbalanced problems than standard algorithms with resampling. However, they may be more computationally expensive or harder to interpret.
Choosing a Technique
With all these techniques available, which one should you use for your imbalanced classification problem? The answer, as with most things in machine learning, is that it depends.
Some factors to consider:
- Dataset size: If your training set is very large, resampling techniques may be too computationally expensive, and adjusting class weights or using special algorithms may be better options.
- Need for interpretable models: Some resampling techniques, like SMOTE, generate synthetic examples that may be hard to understand. If model interpretability is important, adjusting class weights or using interpretable algorithms like decision trees may be preferable.
- Generalizing to new data: Oversampling techniques risk overfitting to the minority class, while undersampling may discard useful information in the majority class. Be sure to thoroughly evaluate your model on an independent test set that reflects the real data imbalance.
In practice, the best approach is often to try multiple techniques and see which one performs best on your particular dataset and problem. Just be sure to use appropriate evaluation metrics to assess performance.
Conclusion
Imbalanced datasets are a common challenge in real-world classification problems. Standard algorithms often fail on such datasets, biasing towards the majority class and ignoring the important minority class.
In this post, we explored five techniques for handling imbalanced datasets:
- Resampling the dataset by oversampling the minority class or undersampling the majority class
- Synthesizing new minority class examples using techniques like SMOTE
- Adjusting class weights to penalize misclassification of the minority class more strongly
- Using appropriate evaluation metrics that account for class imbalance, like precision, recall, and F1 score
- Using specialized learning algorithms designed for imbalanced data
Each technique has its pros and cons in terms of computational cost, risk of overfitting or information loss, and model interpretability. The best approach for a given problem will depend on the characteristics of the dataset and the goals of the analysis.
The key takeaways are:
- Always check your data for class imbalance and consider it when selecting algorithms and evaluation metrics
- Try multiple techniques and see which one performs best on your dataset
- Use appropriate evaluation metrics that reflect your goals and the real cost of different types of errors
- Be wary of overfitting and test your model‘s generalization performance on real imbalanced data
With these tools in your machine learning toolkit, you‘ll be well equipped to tackle imbalanced classification problems and build models that work well on real-world data. Happy classifying!
Further Reading
- Imbalanced-learn documentation: https://imbalanced-learn.org/
- "SMOTE: Synthetic Minority Over-sampling Technique" (research paper): https://arxiv.org/pdf/1106.1813.pdf
- "Learning from Imbalanced Data" (survey paper): https://www.ele.uri.edu/faculty/he/PDFfiles/ImbalancedLearning.pdf
