Introduction to Ensembling: Combining Models for Powerful Machine Learning

Ensemble learning has become one of the most powerful and commonly used techniques in applied machine learning. Whether it‘s winning Kaggle competitions, achieving state-of-the-art results in research, or driving billions of dollars in revenue at companies like Google and Microsoft, ensemble methods are at the forefront of the field.

At its core, the idea of ensembling is to combine multiple models together to produce one unified, superior output. As the famous statistician George E.P. Box said, "All models are wrong, but some are useful." By combining many different models, ensemble techniques can help alleviate the deficiencies of any single model and leverage their collective strengths.

Consider a real-world analogy of a patient visiting multiple specialist doctors, each of whom has expertise diagnosing different conditions. A patient with a complex array of symptoms may receive somewhat varying opinions from the different doctors. But by considering the assessments collectively, the patient is likely to reach a more accurate unified diagnosis than just listening to any single doctor. This same concept underlies ensemble learning.

A more technical explanation is provided by Zhi-Hua Zhou in his seminal book "Ensemble Methods in Machine Learning":

"The basic idea of ensemble learning is to construct multiple learners and combine their predictions. The generalization ability of an ensemble is usually significantly better than that of a single learner, so ensemble learning has been a hot topic in recent years."

Some of the most popular and successful machine learning models today are ensembles. Random forest, which combines many individual decision trees via bagging, and boosted trees, which iteratively combines trees in sequence, are used in a wide array of applications. Many of the winning solutions to data science competitions make heavy use of ensembling.

One prominent example is the Netflix Prize competition, which sought to improve the company‘s movie recommendation algorithm. The winning solution, which claimed the $1 million grand prize, was an ensemble that combined over 100 different models. This demonstrates the power of ensembles to achieve performance gains even with many weak individual learners.

Statistics back up the widespread usage and success of ensembling:

• In a 2020 Kaggle survey of data scientists, 76% of respondents reported regularly using ensemble methods
• A study of winning solutions across 4 years of Kaggle competitions found that ensembles were used in 93% of the winning submissions
• A paper titled "Do We Need Hundreds of Classifiers to Solve Real World Classification Problems?" analyzed 121 datasets and found that a boosted decision tree ensemble achieved the best performance on the majority

Bagging, Boosting, and Stacking

There are three primary classes of ensemble techniques that have proven useful across many domains: bagging, boosting, and stacking. While similar at a high level in that they all combine individual model outputs, they differ in the specifics of their methodology.

Bootstrap Aggregation (Bagging)

Bagging, which is short for bootstrap aggregating, involves training multiple instances of a model on different subsets of the training data. The different subsets are formed by repeatedly sampling observations from the original training set with replacement, which is known as bootstrapping.

The key idea is that by training models on different bootstrapped subsets, the models will become more diverse and make somewhat different errors. When the models‘ predictions are combined, often by simply averaging or majority voting, the errors are averaged out and performance improves.

Mathematically, bagging can be shown to reduce the variance of the predictions. If we assume each of the $n$ models makes prediction $f_i(x)$ for an input $x$ and has variance $\sigma^2$, the variance of the average prediction is:

$$Var \left(\frac{1}{n}\sum_{i=1}^n f_i(x)\right) = \frac{\sigma^2}{n}$$

As the number of models $n$ increases, the variance of the combined prediction decreases, making bagging especially effective for high-variance, low-bias models like deep decision trees.

The most famous bagging ensemble is the random forest, which combines many decorrelated decision trees. Random forests use an additional layer of randomness beyond just bootstrapping the data – they also randomly sample a subset of features to use at each split point in the trees. This increases the diversity of the trees and has been empirically shown to improve performance over standard bagged trees in most cases.

Boosting

In contrast to bagging‘s parallel training of independent models, boosting works by sequentially combining a series of weak models in a greedy, iterative fashion. Each additional model is trained to focus on the examples that the previous models struggled with.

The general procedure for boosting is:

1. Train an initial model
2. Increase the weights of examples the model made errors on
3. Train a new model on the weighted examples
4. Repeat steps 2-3 for a specified number of iterations
5. Combine all the models‘ predictions, weighting later models more heavily

Some of the most common and successful boosting algorithms include AdaBoost, gradient boosting machines (GBMs), and XGBoost. Boosting has been used to great effect in applications as diverse as web search ranking, financial risk modeling, and particle physics.

Boosting is backed up by solid mathematical theory. The original AdaBoost paper by Freund and Schapire shows that the training error of the combined classifier decreases exponentially with the number of iterations, as long as each new classifier has error less than 50%. For binary classification with labels $y_i \in {-1, 1}$, they prove that after $T$ iterations, the training error is bounded by:

$$\frac{1}{N} \sum{i=1}^N \mathbf{1}{H(x_i) \neq yi} \leq \exp \left( -2 \sum{t=1}^T \gamma_t^2 \right) $$

where $\gamma_t$ is the weighted accuracy of the classifier at iteration $t$ and $H(x)$ is the combined classifier. So as long as each new classifier has accuracy better than random guessing, the training error decreases exponentially! This provides strong theoretical justification for the effectiveness of boosting.

Stacking

The third major class of ensemble techniques is stacking, short for stacked generalization. Unlike bagging and boosting which combine similar models in parallel or sequence, stacking combines a diverse set of different models in a layered architecture.

The key idea is to use the outputs of a collection of base models as the inputs to a higher-level meta-model. The base models are trained on the original features, then the meta-model is trained on the outputs of the base models to produce the final predictions.

Mathematically, if we have $K$ base models which produce outputs $f_i(x), i=1 \dots K$, the stacked model computes:

$$H(x) = g(f_1(x), f_2(x), \dots, f_K(x))$$

where $g$ is the meta-model, usually a simple linear model or sometimes a neural network. The intuition is that the meta-model can learn the strengths and weaknesses of the different base models and how to optimally combine their predictions.

Netflix used a stacked ensemble for their final winning submission in the famous Netflix Prize competition. They combined a large collection of diverse models including matrix factorization, restricted Boltzmann machines, and linear regression via a meta neural network.

One nuance that‘s important for stacking is that the outputs of the base models on the training set should not be used as inputs to fit the meta-model on that same training set. This would lead to overfitting, as the base models can just memorize the training set labels. Instead, the base models should be evaluated on hold-out data not seen during their training to produce the meta-model training set.

Key Takeaways for Using Ensemble Methods

With a solid understanding of the main classes of ensemble methods, there are a few key principles and best practices to keep in mind when using ensembles:

1. Diversity of base models is important. Combining a collection of identical or very similar models is not likely to offer much benefit over just using a single model. Using different algorithms, input features, and/or hyperparameter settings helps create a diverse set of base models.

2. Ensemble size is a double-edged sword. Having more models in an ensemble generally improves performance, but with diminishing returns. At a certain point the computational and memory costs outweigh the marginal performance benefits. An ensemble of 5-10 diverse models often strikes a good balance.

3. Use proper validation. With the added complexity of ensembles also comes more opportunity for mistakes. It‘s critical to validate model performance on completely held-out test data. When doing stacking in particular, the base models‘ predictions used to fit the meta-model must be generated via cross-validation or a separate validation set not seen during base model training.

4. Beware reduced interpretability. While ensembles can provide a significant boost in predictive performance, this often comes at the cost of reduced model interpretability. Combining many different models, each of which may be complex on its own, makes it much harder to understand how the ensemble arrives at its predictions. There are model explanation techniques like SHAP values that can help, but the choice to use an ensemble is often a tradeoff between accuracy and interpretability.

5. Automate where possible. Modern machine learning libraries have made building ensembles much easier than it used to be. Packages like scikit-learn, h2o, and mlr have built-in functionality for defining and training ensemble models. When working with large datasets and many models, automating the training and evaluation process as much as possible helps avoid mistakes.

Conclusions

We‘ve covered a lot of ground in this overview of ensemble learning. The key points to remember are:

• Ensembles combine the predictions of multiple individual models to produce a unified output
• Bagging trains models in parallel on bootstrapped samples and takes a simple average/vote
• Boosting iteratively trains models in sequence, upweighting examples that previous models got wrong
• Stacking trains a higher-level meta-model on the outputs of base models
• Diversity of base models is important for ensemble performance
• Proper validation is critical when building ensembles to avoid overfitting
• Ensembles can improve performance but often at the cost of interpretability

Ensembles offer a powerful and widely-applicable tool for machine learning practitioners to get the most out of their models. While they are not a silver bullet, they have proven their value time and time again across a wide variety of domains and problem types. When used properly with careful design and validation, ensembles can provide a reliable way to boost predictive performance.

Additional Resources

If you‘d like to dive deeper into ensemble methods, here are some additional high-quality resources to check out:

• Ensemble Methods in Machine Learning (Book chapter by Zhi-Hua Zhou)
• Ensemble Machine Learning: Methods and Applications (Book by Cha Zhang and Yunqian Ma)
• Kaggle Ensembling Guide (Detailed blog post with lots of practical advice)
• Understanding ML Ensembles (Article explaining key ensemble concepts with code examples)
• Data Science Bowl 2017 Winner‘s Interview (Writeup of solution that used many ensembles)

Here are a few excerpts of advice from Kaggle competition grandmasters on their ensemble usage:

"In general, the goal is to have models that are as diverse as possible while still being individually strong. One way to do this is to use different algorithms for the same problem, e.g. combining gradient boosting with a neural net. Another approach is to use the same algorithm but vary the input features and/or model hyperparameters significantly." (Gilberto Titericz)

"The power of ensembling is that different models can pick up on different patterns, and the diversity helps the combined prediction. I usually try a bunch of different models and then start combining them in a stacking fashion. I also like to blend multiple stacking layers together." (KazAnova)

"For the Netflix Prize, our final ensemble contained over 100 different models. The models were a combination of collaborative filtering, singular value decomposition, restricted Boltzmann machines, and others. We found that having a large number of diverse models was key to getting the best performance." (BellKor team)

Clearly, top data scientists make extensive use of ensemble methods when building state-of-the-art models. While building a blend of 100 models may be overkill for most applications, the general concepts of model diversity and stacking are widely useful. Whenever working on a prediction problem, think about whether you can combine multiple models together to eke out better performance.