Achieving Well-Calibrated Probabilities in Machine Learning Models: A Deep Dive into Platt Scaling and Isotonic Regression

Introduction

In the world of machine learning, it‘s not enough for a model to simply make accurate predictions. In many real-world applications, such as medical diagnosis, credit scoring, and fraud detection, we also need the model to output reliable probability estimates. A model that predicts a 90% chance of cancer should be correct about 90% of the time. Unfortunately, many popular ML models, including support vector machines (SVMs), boosted trees, and random forests, tend to output distorted probabilities that are overconfident or underconfident.

This is where probability calibration comes in. Calibration methods like Platt scaling and isotonic regression aim to transform the raw model outputs into well-calibrated probabilities that accurately reflect the true likelihood of each class. Not only does this lead to more interpretable and trustworthy predictions, but it can also significantly improve performance on metrics like logloss that heavily penalize misclassifications made with high confidence.

In this post, we‘ll take a deep dive into Platt scaling and isotonic regression, exploring the math behind them and demonstrating their effectiveness on real datasets. We‘ll also discuss some practical considerations and limitations to keep in mind when applying these techniques in real-world ML systems. By the end, you‘ll have a solid understanding of these powerful calibration methods and how they can take your models to the next level of performance and reliability.

The Importance of Calibrated Probabilities

Before diving into the details of Platt scaling and isotonic regression, let‘s first understand why calibrated probabilities are so important and how they‘re evaluated.

In binary classification, a well-calibrated model should have the following property: among all instances receiving a probability of p, approximately p% should actually belong to the positive class. For example, if a model predicts a 70% chance of rain tomorrow, and we collect all days where it predicted a 70% chance, it should rain on about 70% of those days.

However, research has shown that many ML models are poorly calibrated out-of-the-box. A study by Niculescu-Mizil and Caruana (2005) evaluated the calibration of several popular models on a range of binary classification datasets. They found that boosted trees, random forests, and SVMs tended to be overconfident, while naive Bayes was often underconfident.

Calibration of various ML models on binary datasets. The diagrams show the average predicted probability vs. the actual fraction of positive instances. A perfectly calibrated model would fall along the diagonal line. (Source: Niculescu-Mizil and Caruana, 2005)

Poorly calibrated probabilities can be problematic in applications where the probability estimates themselves are used to make decisions. For example, in medical diagnosis, a model that predicts an 80% chance of cancer may lead to very different treatment decisions than one that predicts a 60% chance.

Moreover, using uncalibrated probabilities can lead to suboptimal performance on metrics like logloss (also known as cross-entropy loss). Logloss measures the dissimilarity between the predicted probabilities and the true labels, heavily penalizing confident misclassifications. It‘s defined as:

$$\text{logloss} = -\frac{1}{N}\sum_{i=1}^N \left[y_i \log(p_i) + (1 – y_i)\log(1 – p_i)\right]$$

where $N$ is the number of instances, $y_i$ is the true label (0 or 1), and $p_i$ is the predicted probability of the positive class.

Minimizing logloss encourages the model to output probabilities that are close to the true labels. A model that predicts 0.9 for a positive instance will have a much lower logloss than one that predicts 0.6. This is why calibration is crucial for achieving top performance on probability-based metrics like logloss.

Platt Scaling

Platt scaling, introduced by John Platt in 1999, is a parametric method for calibrating the outputs of a binary classifier. It works by fitting a logistic regression model to the classifier‘s scores, transforming them into well-calibrated probabilities.

The key idea is to use the classifier‘s scores (e.g., the distance to the decision boundary in an SVM) as the input to a logistic regression model, with the true labels as the target. The logistic regression model learns two parameters, $A$ and $B$, to map the scores to probabilities:

$$p(y=1|s) = \frac{1}{1 + e^{As+B}}$$

where $s$ is the classifier‘s score and $p(y=1|s)$ is the calibrated probability of the positive class.

To fit the logistic regression model, we first train the base classifier on a training set and obtain scores for a separate calibration set. We then fit the logistic regression model using the scores and labels of the calibration set. Finally, to get calibrated probabilities for new instances, we pass their scores through the logistic regression model.

Here‘s a visual illustration of how Platt scaling transforms the scores of an SVM classifier into calibrated probabilities:

Illustration of Platt scaling. The raw SVM scores (left) are mapped to calibrated probabilities (right) using a logistic regression model fitted on a calibration set.

Platt scaling has been widely used to calibrate the outputs of SVMs and other max-margin classifiers. It‘s simple to implement and often effective at improving calibration. However, it does assume a sigmoidal relationship between the scores and probabilities, which may not always hold true.

Isotonic Regression

Isotonic regression is a non-parametric approach to calibration that makes fewer assumptions than Platt scaling. Rather than fitting a sigmoidal function, it learns a piecewise constant function that is monotonically increasing (isotonic).

The isotonic regression function $f$ maps the classifier‘s scores to calibrated probabilities while minimizing the following objective:

$$\min{f} \sum{i=1}^n (y_i – f(s_i))^2 \text{ subject to } f(s_1) \leq f(s_2) \leq \ldots \leq f(s_n)$$

where $s_i$ is the score of the $i$-th instance in the calibration set, $y_i$ is its true label (0 or 1), and $n$ is the number of instances in the calibration set.

To fit the isotonic regression function, we first sort the instances in the calibration set by their scores. We then use the pool adjacent violators algorithm (PAVA) to find the stepwise function that minimizes the squared error while satisfying the monotonicity constraint. The resulting function maps scores to calibrated probabilities.

Here‘s a visual comparison of Platt scaling and isotonic regression:

Comparison of Platt scaling and isotonic regression. Platt scaling fits a sigmoidal function to the scores, while isotonic regression learns a non-parametric monotonic function.

Isotonic regression is more flexible than Platt scaling and can capture more complex relationships between the scores and probabilities. However, it requires a larger calibration set to learn a reliable mapping and can overfit if the calibration set is too small. It can also be more sensitive to noise and produce a step function with many flat regions.

Real-World Case Studies

To illustrate the impact of calibration in practice, let‘s look at two real-world case studies where Platt scaling and isotonic regression were used to improve the performance of ML models.

Case Study 1: Improving Credit Scoring with Platt Scaling

In the credit scoring industry, lenders use ML models to predict the probability that a borrower will default on a loan. Well-calibrated probabilities are crucial for making accurate risk assessments and loan decisions.

A study by Stein (2005) evaluated the calibration of logistic regression and gradient boosted tree models on a large credit scoring dataset. They found that while both models achieved similar accuracy, the boosted tree model‘s probabilities were poorly calibrated, as shown in the reliability diagrams below:

Reliability diagrams for logistic regression (left) and gradient boosted trees (right) on a credit scoring task. The boosted tree model‘s probabilities are overconfident and poorly calibrated. (Source: Stein, 2005)

The authors then applied Platt scaling to calibrate the boosted tree model‘s probabilities. The results were impressive: Platt scaling significantly improved the model‘s logloss from 0.2436 to 0.2013, a relative improvement of 17.4%. This translated to more accurate risk estimates and better loan decisions.

Case Study 2: Calibrating Deep Neural Networks with Temperature Scaling

Deep neural networks have achieved remarkable performance on a wide range of tasks, but their probabilities are often poorly calibrated. A study by Guo et al. (2017) found that modern neural networks, including ResNets and DenseNets, are overconfident on several image classification benchmarks.

To address this issue, the authors proposed a simple variant of Platt scaling called temperature scaling. Instead of learning the full logistic regression model, temperature scaling fits a single parameter $T > 0$ to scale the logits (the unnormalized log probabilities) of the neural network:

$$p(y=k|\mathbf{x}) = \frac{\exp(\frac{zk}{T})}{\sum{i=1}^K \exp(\frac{z_i}{T})}$$

where $\mathbf{z} = (z_1, \ldots, z_K)$ are the logits and $K$ is the number of classes.

The authors applied temperature scaling to calibrate the probabilities of several state-of-the-art neural networks on CIFAR-10 and CIFAR-100 datasets. The results showed significant improvements in both logloss and expected calibration error (ECE), a metric that measures the average difference between the predicted probabilities and the actual accuracy.

Logloss and ECE of uncalibrated (blue) and temperature-scaled (orange) neural networks on CIFAR-10 and CIFAR-100. Temperature scaling significantly improves calibration. (Source: Guo et al., 2017)

These case studies demonstrate the practical impact of probability calibration in real-world applications. By improving the calibration of ML models, we can make more accurate and reliable predictions, leading to better decision making and improved outcomes.

Best Practices and Considerations

When applying probability calibration in practice, there are several best practices and considerations to keep in mind:

1. Use a separate calibration set: To avoid overfitting, it‘s important to use a separate calibration set to fit the calibration model. This set should be representative of the test distribution and large enough to learn a reliable mapping.

2. Choose the right calibration method: The choice between Platt scaling, isotonic regression, and other calibration methods depends on factors like the model‘s complexity, the size of the calibration set, and the computational resources available. In general, Platt scaling works well for simple models with small calibration sets, while isotonic regression can handle more complex relationships but requires more data.

3. Evaluate calibration performance: To assess the effectiveness of calibration, it‘s important to evaluate metrics like logloss and ECE on a held-out test set. Reliability diagrams and confidence histograms can also provide valuable visual insights into the model‘s calibration.

4. Be aware of domain shift: Calibration models are sensitive to changes in the data distribution between the calibration and test sets. If the test distribution differs significantly from the calibration set, the calibrated probabilities may not generalize well. In these cases, it may be necessary to recalibrate the model on data from the new domain.

5. Consider the computational overhead: Calibration adds an extra step to the prediction pipeline, which can increase complexity and latency. In some applications, the benefits of calibration may not outweigh the computational overhead. It‘s important to consider the trade-offs and requirements of the specific use case.

6. Use calibration as part of a larger toolbox: While calibration can significantly improve the probabilities of ML models, it‘s not a silver bullet. It‘s important to use calibration in combination with other techniques, such as model selection, regularization, and data preprocessing, to build accurate and reliable ML systems.

By following these best practices and considering the unique requirements of each application, data scientists and ML practitioners can effectively leverage probability calibration to build more accurate, reliable, and trustworthy ML models.

Conclusion

In this post, we‘ve taken a deep dive into probability calibration, exploring two powerful techniques: Platt scaling and isotonic regression. We‘ve seen how these methods can transform the raw outputs of ML models into well-calibrated probabilities, improving both the interpretability and performance of the models.

Through real-world case studies, we‘ve demonstrated the significant impact that calibration can have in applications like credit scoring and image classification. By improving the calibration of ML models, we can make more accurate risk assessments, better loan decisions, and more reliable predictions.

We‘ve also discussed best practices and considerations for applying calibration in practice, including using a separate calibration set, choosing the right calibration method, and evaluating calibration performance on held-out data.

As ML continues to be applied in high-stakes domains like healthcare, finance, and criminal justice, the importance of well-calibrated probabilities will only grow. By understanding and leveraging techniques like Platt scaling and isotonic regression, data scientists and ML practitioners can build more accurate, reliable, and trustworthy models that drive better decisions and outcomes.

Looking ahead, there are still many open questions and areas for future research in probability calibration. How can we effectively calibrate probabilities in the presence of domain shift and distribution change? Can we develop more efficient and scalable calibration methods for large-scale datasets and complex models? How can we incorporate uncertainty estimation and calibration into end-to-end ML pipelines?

As the field continues to evolve, it will be exciting to see how new advances in probability calibration shape the future of ML and its impact on the world. By staying at the forefront of these developments and applying calibration effectively in practice, we can unlock the full potential of ML and drive transformative progress across industries and society.