Calibration of Machine Learning Models: Making Your Model Probabilities Reliable
When building machine learning models, we usually focus on metrics like accuracy, precision, and recall to measure their performance. However, there is another crucial aspect that is often overlooked – the calibration of the model‘s predicted probabilities. A model is considered well-calibrated if the predicted probabilities match the actual likelihood of the events. For example, if a model predicts a 70% chance of rain on 100 different days, we expect it to actually rain on about 70 of those 100 days.
Calibration is critical for any application where the model‘s predicted probabilities are directly used in decision making, such as weather forecasting, medical diagnosis, financial risk assessment, and fraud detection. Using an uncalibrated model in these scenarios can lead to suboptimal decisions and negative consequences. A lender may approve too many risky loans if the model is underestimating default probabilities. A self-driving car may make dangerous maneuvers if it is overconfident in its object detection.
In this post, we‘ll dive deep into model calibration – why it matters, how to measure and visualize it, methods to calibrate models, best practices, and limitations to be aware of. By the end, you‘ll have a solid grasp of calibration and how to integrate it into your machine learning workflow to make your models more reliable and trustworthy. Let‘s get started!
Visualizing Calibration with Reliability Diagrams
The most common way to assess calibration visually is using a reliability diagram, also known as a calibration plot. It plots the predicted probabilities on the x-axis against the actual fraction of positive instances on the y-axis. To construct it, the predicted probabilities are divided into bins (e.g. 0-0.1, 0.1-0.2, etc.) and the true positive rate is calculated for the instances in each bin.
If the model is perfectly calibrated, the points will fall along the diagonal line, indicating that the predicted probabilities match the actual likelihoods. Deviations from the diagonal reveal miscalibration. If the points fall below the diagonal, it means the model is overconfident – the predicted probabilities are higher than the true likelihoods. Points above the diagonal indicate underconfidence.
Here‘s an example reliability diagram for an uncalibrated model:
We can see that this model is quite miscalibrated, with the curve deviating substantially from the ideal diagonal line. At the lower end, the model is underconfident, with the actual likelihood being higher than predicted. At the higher probabilities, the model becomes overconfident.
Reliability diagrams provide valuable insights into the calibration across the probability range. They can reveal if the miscalibration is occurring only at certain probabilities or if it‘s a systematic issue. However, they don‘t give a single quantitative metric to compare different models. For that, we‘ll look at some scoring functions next.
Quantifying Calibration with Scoring Metrics
To compare the calibration of different models objectively, we need quantitative metrics. Some commonly used ones are:
Brier Score: This is the mean squared difference between the predicted probabilities and the actual outcomes (0 or 1). It ranges from 0 to 1, with lower scores indicating better calibration. Brier score can be decomposed into calibration loss and refinement loss terms.
Expected Calibration Error (ECE): ECE calculates the average absolute difference between the predicted probabilities and actual likelihoods in the reliability diagram bins, weighted by the number of instances in each bin. It is a more intuitive metric, with 0 being perfect calibration and higher values indicating worse calibration.
Maximum Calibration Error (MCE): MCE is the maximum absolute difference between predicted and actual probabilities across the bins. It measures the worst-case deviation from perfect calibration.
Here‘s how they are calculated:
where N is the total number of instances, M is the number of bins, Bm is the set of indices of instances in bin m, |Bm| is the number of instances in bin m, pi is the predicted probability for instance i, and yi is the actual outcome (0 or 1).
These metrics give us a way to quantify calibration and compare models. However, it‘s important to note that they are affected by the choice of binning scheme. Using too few bins can hide miscalibration, while too many bins can make the estimates noisy, especially with small datasets. In practice, 10-15 equal-width bins are commonly used.
Methods for Calibrating Models
If our model is miscalibrated, how can we fix it? There are two main approaches: Platt scaling and isotonic regression. Both of these are post-processing methods that adjust the predicted probabilities without changing the underlying model.
Platt Scaling:
Platt scaling fits a logistic regression model on the logit of the predicted probabilities. The idea is to learn a simple transformation that maps the uncalibrated probabilities to calibrated ones. The logistic function has the nice property of outputting well-calibrated probabilities.
The scaling is done by training on a separate calibration dataset, different from the one used to train the original model. This is important to avoid overfitting. The scaled probabilities p‘ are given by:
where p is the original predicted probability and A, B are the learned parameters of the logistic regression.
Platt scaling works well when the uncalibrated reliability diagram has a sigmoid-shaped curve. It is a simple and fast method that often gives good results with minimal tweaking. However, it assumes a parametric form and may not work well if the curve has a different shape.
Isotonic Regression:
Isotonic regression is a more flexible, non-parametric approach. Instead of assuming a sigmoid form, it fits a piecewise constant monotonically increasing function to the data. The idea is to find the isotonic (monotonically increasing) function that minimizes the mean squared error on the calibration dataset.
The isotonic regression function is learned by fitting the pooled predicted probabilities against the actual outcomes. It outputs a mapping from the original probabilities to the calibrated ones.
Isotonic regression can handle any monotonic shape of the uncalibrated curve. It is more powerful than Platt scaling but requires a larger calibration dataset to avoid overfitting, since it has more flexibility. It is also more computationally intensive.
In practice, the choice between Platt scaling and isotonic regression depends on factors like the size of the calibration dataset, the shape of the uncalibrated curve, and computational constraints. It‘s recommended to try both and choose the one that gives the best results on a held-out validation set.
Limitations and Considerations
While calibration is important, it‘s not a silver bullet. Here are some limitations and things to keep in mind:
Calibration-Discrimination Trade-off: In some cases, the process of calibrating a model can slightly reduce its discrimination ability, i.e. how well it ranks instances. This is known as the calibration-discrimination trade-off. In applications where ranking is more important than absolute probabilities, calibration may not be desirable.
Uncertainty of Calibration: The reliability of the calibrated probabilities depends on the size and representativeness of the calibration dataset. With small calibration sets, there can be uncertainty in the calibration estimate. Confidence intervals around the calibration curve can be used to assess this uncertainty.
Stability of Calibration: The calibration of a model can change over time if the data distribution shifts. It‘s important to monitor the calibration on new data and recalibrate the model if necessary.
Multiclass Calibration: Calibrating multiclass models is more challenging than binary ones. Extension of binary calibration methods to multiclass often gives poor results. Specialized methods like matrix scaling or Dirichlet scaling are needed.
Calibration ≠ Accuracy: A well-calibrated model is not necessarily a highly accurate model. Calibration measures the reliability of the predicted probabilities, not the correctness of the predictions. It‘s possible to have a well-calibrated model with low accuracy, and vice versa. Calibration and accuracy are complementary, and both are important for a good model.
Best Practices for Model Calibration
To get the most out of calibration, here are some recommendations and best practices:
-
Always use a separate calibration dataset, different from the training and testing data. This prevents overfitting and gives a more reliable estimate of the calibration.
-
Choose the calibration method (Platt scaling or isotonic regression) based on empirical results on a validation set. Don‘t assume one is always better than the other.
-
Assess calibration using multiple metrics (Brier score, ECE, MCE) and reliability diagrams. Relying on a single number can be misleading.
-
Recalibrate the model periodically if the data distribution is expected to change over time. Calibration is not a one-time process.
-
Be cautious when calibrating small datasets. The uncertainty in the calibration estimate can be high, and over-calibration can lead to worse performance.
-
For multiclass problems, use specialized calibration methods designed for multiclass setting, rather than naively extending binary methods.
-
Don‘t use calibration as a band-aid for a weak model. Aim to improve the accuracy and discrimination ability of the model first, then calibrate it to get reliable probabilities.
Conclusion
We have covered a lot of ground in this post, diving deep into the concept of model calibration. To summarize:
- Calibration measures the reliability of a model‘s predicted probabilities. A well-calibrated model outputs probabilities that match the actual likelihoods of the events.
- Calibration is important when the predicted probabilities are used for decision making. Miscalibrated models can lead to suboptimal decisions.
- Reliability diagrams provide a visual way to assess calibration. Quantitative metrics like Brier score, ECE, and MCE can be used to compare models.
- Platt scaling and isotonic regression are two popular methods for calibrating models. They adjust the predicted probabilities without changing the underlying model.
- Calibration has some limitations and trade-offs to consider, such as reduced discrimination, uncertainty, and stability over time.
- Following best practices like using a separate calibration set, combining metrics and diagrams, and recalibrating periodically can help get the most out of calibration.
Model calibration is a valuable tool in the machine learning practitioner‘s toolbox. By making the model probabilities more reliable and trustworthy, it enables better decision making and improves the overall utility of the model. I hope this post has given you a comprehensive understanding of calibration and how to incorporate it into your workflow.
As a next step, I encourage you to try out calibration on your own models and see the impact it has. Remember, a well-calibrated model is a reliable model, and reliability is key in any high-stakes application.
Happy calibrating!
