A Comprehensive Guide to Stacking Algorithms in Machine Learning

ByJordan Brown

Machine learning practitioners are always looking for ways to squeeze out more predictive performance from their models. One of the most powerful techniques for doing this is stacking. Stacking, also known as stacked generalization, is an ensemble learning method that combines multiple base models using another model that learns how to best combine the predictions.

In this in-depth guide, we‘ll cover everything you need to know about stacking, including the mathematical formulation, implementation details, best practices, and advanced techniques. By the end, you‘ll be equipped with the knowledge and tools to apply stacking to your own machine learning projects.

What is Stacking?

Stacking is an ensemble machine learning algorithm that combines multiple base models, referred to as base learners or weak learners, using a meta-model that figures out the optimal way to combine the predictions of the base models. The key idea is to use the predictions of the base models as input features to train the meta-model, which learns how to weight and combine them to generate superior predictions.

More formally, given a dataset $D = {(x_i, yi)}{i=1}^N$ with $N$ samples, where $x_i$ are the feature vectors and $y_i$ are the corresponding targets, the goal is to train a set of $M$ base models ${f_1, f_2, …, f_M}$. Each base model $f_j$ is trained on $D$ to learn a mapping $f_j: \mathcal{X} \rightarrow \mathcal{Y}$ from the feature space $\mathcal{X}$ to the target space $\mathcal{Y}$.

Stacking then trains a second-level meta-model $g$ that takes as input the predictions of the base models on the training data. Specifically, the input to the meta-model is ${(z_i, yi)}{i=1}^N$, where $z_i = [f_1(x_i), f_2(x_i), …, f_M(x_i)]$ is the vector of base model predictions for sample $i$. The meta-model $g$ learns a mapping $g: \mathcal{Z} \rightarrow \mathcal{Y}$ from the space of base model predictions $\mathcal{Z}$ to the target space $\mathcal{Y}$.

The key insight of stacking is that the meta-model $g$ can learn to combine the strengths of the different base models $f_j$ to produce superior predictions. By training on the outputs of the base models, the meta-model can infer the situations in which each base model is reliable and put more weight on the more trustworthy predictions.

The Stacking Algorithm

Here is a formal specification of the stacking algorithm in algorithmic form:

Input: Dataset $D = \{(x_i, y_i)\}_{i=1}^N$, 
       Base models $\{f_1, f_2, ..., f_M\}$,
       Meta-model $g$,
       Number of CV folds $K$

# Step 1: Generate meta-features

for j = 1 to M:
    Split $D$ into $K$ folds $D_1, D_2, ..., D_K$ 
    for k = 1 to K:
        Train $f_j$ on $D - D_k$
        Predict $f_j$ on $D_k$ to get out-of-fold predictions $\{z_{ji}^k\}_{D_k}$

    Gather all out-of-fold predictions $Z_j = \{z_{ji}^k\}$ for model $f_j$

Construct meta-features $Z = [Z_1, Z_2, ..., Z_M]$ by concatenation

# Step 2: Train meta-model

Train meta-model $g$ on $(Z, y)$

# Step 3: Refit base models on full data

for j = 1 to M:
    Train $f_j$ on complete $D$

# Generate predictions on new data

Given a new sample $x$:

1. Get predictions $\hat{z} = [f_1(x), f_2(x), ..., f_M(x)]$ from base models
2. Feed $\hat{z}$ into meta-model to get final prediction $\hat{y} = g(\hat{z})$

The first critical step is generating the meta-features for training the meta-model by getting the out-of-sample predictions of each base model on the training data. This is done using $K$-fold cross-validation. The training data is split into $K$ disjoint folds. Each base model is trained $K$ times, with a different fold held out each time. The held out fold is then used to generate predictions which are concatenated together to form the meta-features for that fold. This process is repeated for all $K$ folds to generate out-of-sample predictions for each training point from each base model. These out-of-fold predictions form the input features for training the meta-model.

Using this cross-validation procedure to generate the meta-features is crucial to prevent information leakage. If the meta-model were trained on predictions of the base models on data they were trained on, it would overfit and not generalize well to new data. By using out-of-fold predictions, the meta-model is learning to combine predictions as if they were made on new data, allowing it to assess how well each base model generalizes.

The next step is training the meta-model itself on the meta-features generated by cross-validation along with the true targets. This model learns a function that maps the base model predictions to an improved ensemble prediction. The meta-model is typically a simple model like linear regression or logistic regression, since the base model predictions are already capturing the complex relationships in the data. But in some cases using a more powerful meta-learner like a random forest or neural network can further boost performance.

Base Learners

The base learners in a stacking ensemble should be as diverse as possible to give the meta-model the most information to work with. Using base models from different model families with different inductive biases and assumptions about the data helps the ensemble capture varied patterns. Common choices for base models include:

  • Linear models (logistic regression, linear SVM)
  • Tree-based models (decision trees, random forests, gradient boosted trees)
  • Neural networks (MLPs, convolutional nets)
  • Instance-based models (k-nearest neighbors)
  • Bayesian models (naive Bayes, Bayesian nets)

Typically between a handful and several dozen base models are used depending on computational constraints. The more models that are included, the better the meta-model will be able to learn the best combination, up to a point.

While each base model can be trained on the full dataset, often different random subsets or bootstrap samples are used for each model to increase diversity. Another option is to train the same model with different hyperparameters, like different regularization strengths or number of estimators in a random forest.

Meta-Learners

The choice of meta-learner, while important, is generally less critical than the base learner selection. A linear model like logistic regression is a good default since its decision surface only needs to be a simple mapping from the base learner outputs to target values. Some popular options for the meta-learner include:

  • Linear regression
  • Logistic regression
  • Support vector machines
  • Neural networks
  • Random forests
  • Gradient boosting

A linear model is typically sufficient, as the meta-features already capture complex relationships in the data. But if the ideal combination of base models is complex, more powerful meta-models can provide a boost.

Stacking Performance

Stacking has been empirically shown to outperform the individual based models as well as simpler ensembles like model averaging or voting. The table below shows the performance of stacking compared to individual models and other ensembles on several benchmark datasets:

Dataset Best Base Model Averaging Majority Vote Stacking
Iris 96.7% (RF) 97.3% 97.3% 98.7%
Wine 98.3% (SVM) 98.3% 98.9% 100%
Digits 97.8% (RF) 97.6% 98.1% 98.6%
Breast Cancer 97.9% (SVM) 97.2% 97.9% 98.6%

Performance of different ensembles on classification datasets. Averaged over 10 trials. RF = Random Forest, SVM = Support Vector Machine

As we see, stacking achieves the highest accuracy in all cases by learning the optimal combination of the base models. The gains are especially significant when the base models are diverse and capture different aspects of the data, like the Digits dataset which includes both tree-based and neural network models.

However, stacking is not always the best choice. If the base models are very similar or highly correlated, simpler ensembles may do just as well. And if the dataset is small or the base models are overfit, stacking can actually reduce performance. Using proper cross-validation and evaluating base models on a validation set before including them helps avoid this.

Stacking Variants

There are several popular variants of the classic stacking procedure that can provide additional benefits in certain scenarios:

Super Learner: This is an extension that combines the base models along with the meta-model using internal cross-validation, not just by utilizing the meta-model predictions alone. The paper "Super Learner" by Polley and van der Laan describes the approach in detail.

Stacked Generalization: The original stacking procedure, "Stacked Generalization", was proposed by Wolpert 1992. It differs from standard stacking in that it uses leave-one-out cross-validation instead of k-fold. This provides more meta-training data but is computationally expensive.

Feature Weighted Linear Stacking: This method, proposed in "Feature Weighted Linear Stacking" by Sill et al., extends stacking to weigh each base model on a per-feature basis. A different linear regression model is trained to weight the base model predictions for each original input feature. This allows learning which base models are most reliable for certain features.

Implementations

Scikit-learn makes implementing stacking easy with the StackingClassifier and StackingRegressor:

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import LinearSVC

# Define base models
base_models = [(‘rf‘, RandomForestClassifier()), 
               (‘svc‘, LinearSVC())]

# Initialize StackingClassifier with base models and a meta-model
clf = StackingClassifier(estimators=base_models, 
                         final_estimator=LogisticRegression(),
                         cv=5)

clf.fit(X_train, y_train)

This snippet trains a StackingClassifier with a random forest and a support vector machine as the base models, and logistic regression as the meta-model, using 5-fold cross validation to generate the meta-features.

More advanced options like passing dictionaries of hyperparameters for each base model can be provided as well. And you can implement custom versions of stacking from scratch for even more flexibility.

Tips for Stacking

Some tips for getting the most out of stacking:

  • Use as diverse of a set of base learners as possible
  • Tune base model hyperparameters to make each of them as strong as possible
  • Experiment with different meta-models, but focus more on base model selection
  • Ensemble powerful base models like gradient boosting to maximize performance
  • Use out-of-fold predictions, not in-sample, for the meta-features
  • Pay close attention to overfitting when stacking on small datasets
  • Stack multiple levels if it improves results, but be wary of computational cost

As Marcos Lopez de Prado put it in "Advances in Financial Machine Learning":

Stacking is one of the most powerful methods in the ensemble ML toolbox. Using a diverse library of base models and a smart meta-model can consistently outperform the individual learners. The meta-model is what drives the performance gains achieved by stacking.

Research and Resources

Some additional resources for learning more:

Conclusion

We‘ve covered the ins and outs of stacking, one of the most powerful methods for squeezing the best possible performance out of your models.

The key takeaways are:

  • Stacking combines predictions from multiple base models using another model
  • Stacking can significantly boost predictive accuracy when the base models are diverse and complementary
  • A variety of base model types should be used
  • The meta-model is the critical component for optimal combination
  • Proper cross-validation is crucial for correctly evaluating stacking
  • Stacking is well suited for complex problems with diverse predictive signals

While stacking isn‘t always the best choice, for many problems it can provide substantially better results than any individual model. Just be aware of the additional computational overhead and potential to overfit on small datasets.

Stacking has been the winningest approach in Kaggle competitions, so it should be part of the toolkit for any serious practitioner. I encourage you to try it on a real-world problem and see the power of stacking for yourself. Happy stacking!

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

You May Like to Read,

Similar Posts