Measuring Bias and Variance in Machine Learning Models: An In-Depth Guide

As machine learning continues to advance and find applications in an ever-growing range of domains, it‘s more important than ever to understand the factors that influence a model‘s performance. Two of the most fundamental concepts in this regard are bias and variance. In this post, we‘ll take a deep dive into what these concepts mean, why they matter, and how we can measure them through carefully designed experiments.

The Bias-Variance Decomposition

At the core of understanding model performance is the bias-variance decomposition. This decomposition breaks down a model‘s generalization error into three components:

1. Bias: This is the error caused by the model‘s assumptions and simplifications. A high-bias model tends to underfit the data, missing relevant patterns.

2. Variance: This is the error caused by the model‘s sensitivity to small fluctuations in the training data. A high-variance model tends to overfit, memorizing noise in the data.

3. Irreducible Error: This is the error caused by noise in the data itself, which can‘t be reduced by any model.

Mathematically, we can express the expected generalization error of a model f as:

$E[(y – f(x))^2] = Bias(f(x))^2 + Var(f(x)) + \sigma^2$

where $Bias(f(x)) = E[f(x)] – y$, $Var(f(x)) = E[(f(x) – E[f(x)])^2]$, and $\sigma^2$ is the irreducible error.

The goal in machine learning is to minimize this total error. However, there‘s a tradeoff between bias and variance: reducing one usually comes at the cost of increasing the other. This is known as the bias-variance tradeoff.

Bias and Variance of Common Models

Different types of models have different bias-variance properties. Here‘s a summary of some common model types:

Linear models like linear regression have high bias because they make strong assumptions about the data (linearity), but low variance because they‘re relatively insensitive to noise. On the other hand, more complex models like decision trees and k-NN have low bias because they can capture complex patterns, but high variance because they‘re sensitive to noise.

Neural networks, particularly deep ones, are interesting because their bias and variance can vary widely depending on factors like the network architecture, regularization techniques used, and the amount of training data.

Estimating Bias and Variance: An Experiment

To see how we can estimate bias and variance in practice, let‘s walk through an experiment using the Physicochemical Properties of Protein Tertiary Structure dataset. This dataset, from the UCI Machine Learning Repository, contains 45,730 examples of proteins described by 9 physicochemical properties.

Experimental Setup

1. We designate the full dataset of 45,730 examples as our "population."
2. We extract a random subset of 1,500 examples as our test set, leaving 44,230 examples as our training set.
3. We train four types of models (Linear Regression, Decision Tree, Bagging with Decision Trees, and Random Forest) on the full training set and obtain their predictions on the test set. These serve as our "ground truth" predictions.
4. For each model type, we take 30 random samples of the training data at various sizes (100, 500, 1000, 2000, 4000, 8000, 10000), train models on each sample, and obtain their test set predictions.
5. To estimate bias, we calculate the mean absolute difference between the sample models‘ averaged predictions and the ground truth predictions.
6. To estimate variance, we calculate the variance of the sample models‘ predictions.

Results

Here are the bias and variance estimates for each model type at a sample size of 8000:

As expected, linear regression has the lowest bias, suggesting the data is relatively linear. The variance follows the expected pattern, with linear regression lowest and decision trees highest.

Here‘s how the bias and variance change as we increase the sample size:

As the sample size increases, both bias and variance decrease for all model types. However, the improvements become smaller at larger sample sizes.

Cross-Validation

An important note is that these bias and variance estimates are based on a single train-test split. To get more robust estimates, we should use cross-validation. In k-fold cross-validation, we divide the data into k equal parts, use k-1 parts for training and 1 part for testing, and repeat this k times. This gives us k estimates of bias and variance, which we can average.

Cross-validation is especially important when comparing different model types or hyperparameter settings, as it reduces the risk of getting lucky or unlucky with a particular train-test split.

Reducing Bias and Variance

Now that we know how to estimate bias and variance, how can we actually improve our models? Here are some common techniques:

To reduce bias:

• Use a more complex model class (e.g. switch from linear regression to decision trees)
• Add more features to capture more patterns in the data
• Decrease regularization (if using regularized models)

To reduce variance:

• Use a simpler model class
• Gather more training data
• Increase regularization
• Use ensemble methods like bagging or boosting

The optimal choice depends on the specific characteristics of your data and your goals. It‘s often useful to plot the bias and variance of different models as you vary a model complexity parameter (like the maximum depth of a decision tree). This can help you find the "sweet spot" of the bias-variance tradeoff.

The Bias-Variance Tradeoff in Deep Learning

In recent years, the rise of deep learning has led some researchers to question the classical understanding of the bias-variance tradeoff. Deep neural networks are highly complex models with millions or even billions of parameters, which would suggest they should have high variance. Yet in practice, they often generalize very well when trained on large datasets.

One possible explanation is that the bias-variance tradeoff operates differently in the high-dimensional regime of deep learning. When the number of model parameters is much larger than the number of training examples, the model can effectively memorize the training data (high variance) while still finding patterns that generalize (low bias).

Another view is that techniques like early stopping, dropout, and batch normalization implicitly regularize the model, reducing variance. The inductive biases of neural network architectures (e.g. translation invariance in convolutional nets) may also contribute to their ability to generalize despite their complexity.

Conclusion

Bias and variance are fundamental concepts for understanding the behavior and performance of machine learning models. By decomposing a model‘s generalization error into these components, we can diagnose issues like underfitting and overfitting, and make principled choices about model selection and hyperparameter tuning.

Through the experiment with the protein structure data, we saw how we can estimate bias and variance in practice by comparing models trained on samples of the data to a "ground truth" model trained on a large dataset. We observed the expected relationships: simpler models like linear regression tend to have high bias and low variance, while more complex models like decision trees have low bias and high variance.

We also discussed strategies for reducing bias (e.g. using more complex models, adding features) and reducing variance (e.g. regularization, ensembling). The optimal choice depends on the specifics of the problem and the data.

Finally, we touched on some recent research suggesting that the classical understanding of the bias-variance tradeoff may need to be updated in the context of deep learning. As our models become more complex and our datasets larger, new theories may be needed to explain their generalization behavior.

Regardless of these advanced considerations, the bias-variance tradeoff remains a cornerstone concept in machine learning. By keeping it in mind and using techniques like cross-validation to estimate bias and variance, we can diagnose and improve our models, ultimately leading to better performance in real-world applications.