Classification Problems: Understanding Sensitivity, Specificity and Accuracy

Introduction

Classification is one of the most common types of supervised machine learning problems. The goal in classification is to predict which category or class a given input belongs to, based on learning from past labeled examples. Some classic examples include:

• Predicting if an email is spam or not spam
• Diagnosing if a patient has a certain disease or not based on symptoms and test results
• Categorizing images as containing a dog, cat, or some other animal

To build and deploy useful classification models, we need reliable ways to evaluate their performance and understand their strengths and limitations. While overall accuracy is a common metric, it doesn‘t tell the full story. Two other important measures are sensitivity and specificity. In this post, we‘ll take an in-depth look at what these metrics mean, how they‘re calculated, and what they reveal about a classifier‘s behavior.

The Confusion Matrix

The foundation for calculating accuracy, sensitivity and specificity is the confusion matrix. For a binary classification problem, the confusion matrix is a 2×2 table that shows the model‘s predictions versus the actual labels. It gets its name because it makes it easy to see if the model is confusing the two classes.

The four cells of the matrix are:

• True Positives (TP): Input was positive and model predicted positive
• True Negatives (TN): Input was negative and model predicted negative
• False Positives (FP): Input was negative but model predicted positive
• False Negatives (FN): Input was positive but model predicted negative

The confusion matrix forms the basis for most classification evaluation metrics. Let‘s see how accuracy, sensitivity and specificity are defined in terms of the confusion matrix.

Defining Accuracy, Sensitivity and Specificity

Accuracy is the ratio of correct predictions to total predictions. It can be calculated from the confusion matrix as:

Accuracy = (TP + TN) / (TP + TN + FP + FN)

While accuracy gives a good overall sense of the model‘s performance, it can be misleading in imbalanced datasets where one class is much more frequent than the other.

Sensitivity, also known as recall or true positive rate, measures the proportion of actual positive instances that were correctly classified as positive. It is calculated as:

Sensitivity = TP / (TP + FN)

Sensitivity expresses how well the model avoids false negatives. A highly sensitive test rarely misses actual positive cases.

Specificity, also called true negative rate, measures the proportion of actual negative instances that were correctly classified as negative. It is calculated as:

Specificity = TN / (TN + FP)

Specificity expresses how well the model avoids false positives. A highly specific test rarely raises false alarms.

The Sensitivity-Specificity Tradeoff

An important concept to grasp is that sensitivity and specificity are inversely related. Increasing one typically decreases the other. This tradeoff is controlled by the classification threshold – the cutoff value above which the model predicts the positive class.

At a very low threshold, the model predicts almost everything as positive. This maximizes sensitivity but sacrifices specificity. Conversely, a very high threshold makes the model very picky about predicting positive, leading to high specificity but low sensitivity.

We can plot sensitivity and specificity as curves varying over all possible thresholds. The intersection point of these curves indicates the threshold where they are equal. Interestingly, the prediction accuracy is also equal to sensitivity and specificity at this crossing point.

Proof of Equality at Intersection

We can prove mathematically that when sensitivity equals specificity, they also equal accuracy. Let the sensitivity and specificity at the intersection point be denoted by x. Then:

Sensitivity = x = TP / (TP + FN)
Specificity = x = TN / (TN + FP)

Cross-multiplying and rearranging, we get:

TP = x(TP + FN)
TN = x(TN + FP)

Adding these equations:

TP + TN = x(TP + FN + TN + FP)

Dividing both sides by (TP + FN + TN + FP), which is the total number of instances:

(TP + TN) / (TP + FN + TN + FP) = x

The left hand side is exactly the formula for accuracy. Therefore, at the sensitivity-specificity intersection,

Accuracy = Sensitivity = Specificity

Finding the Optimal Threshold

Plotting the sensitivity, specificity and accuracy curves is very useful to visualize their relationships and find a suitable classification threshold for the problem at hand.

Some common strategies for selecting a threshold are:

1. Intersection point of sensitivity and specificity curves (balances both)
2. Prioritize sensitivity or specificity based on the costs of false negatives vs false positives
3. Maximize accuracy or some other combined metric like F1 score

The optimal approach depends on the specific application. In medical diagnosis, high sensitivity is often crucial to avoid missing positive cases. For spam filters, high specificity is desirable to avoid filtering out legitimate emails.

Worked Example

Let‘s solidify these concepts with a concrete example. Consider a model for predicting diabetes based on some biomarkers. We have the following confusion matrix on a test set of 100 patients:

          Predicted
         Neg    Pos
Actual
  Neg     70     5 
  Pos     8     17

Accuracy = (70 + 17) / 100 = 0.87
Sensitivity = 17 / (17 + 8) = 0.68
Specificity = 70 / (70 + 5) = 0.93

The model has high overall accuracy and specificity, but somewhat lower sensitivity. This means it‘s better at ruling out diabetes (high TN) than detecting it (moderate TP). Whether this is acceptable depends on the use case.

If we adjust the threshold to increase sensitivity to 0.8 (same as specificity), the confusion matrix might change to:

          Predicted 
         Neg    Pos
Actual
  Neg     60     15
  Pos     5     20

Now accuracy = sensitivity = specificity = 0.8. This is the equality point we proved mathematically.

Advanced Topics and Extensions

While we focused on binary classification, these concepts extend to multiclass problems as well. Micro- and macro-averaging can be used to calculate overall metrics.

Receiver Operating Characteristic (ROC) curves and precision-recall (PR) curves offer additional tools to analyze the performance of binary classifiers. The area under these curves (AUROC and AUPRC) serve as threshold-independent metrics.

Techniques like stratified sampling and cross-validation are recommended when evaluating classifiers, especially on imbalanced or small datasets, to get more robust estimates of performance.

Recap and Conclusion

• Accuracy, sensitivity and specificity are three key metrics to evaluate classification models, calculated from the confusion matrix
• Sensitivity (true positive rate) and specificity (true negative rate) have an inverse relationship, mediated by the classification threshold
• The intersection point of sensitivity and specificity is where they equal each other and the overall accuracy
• Plotting sensitivity, specificity and accuracy curves helps find optimal thresholds suited to the problem
• Considering domain-specific costs and constraints is important when selecting classification thresholds

We covered a lot of ground in this post, from the basics of confusion matrices to the mathematical relationship of sensitivity, specificity and accuracy. The key takeaway is that a nuanced understanding of these metrics is essential for developing classifiers that are not just accurate, but also sensitive and specific in ways that match the needs of the application.

Proper evaluation is critical to uncover potential biases, limitations and failure modes of classifiers before deploying them. By decomposing performance into multiple views, we can build more robust and reliable AI systems.

Further Reading

• An Introduction to Statistical Learning, Chapter 4: Classification
• Hands-On Machine Learning with Scikit-Learn and TensorFlow, Chapter 3: Classification
• Google Machine Learning Crash Course: Classification Model Evaluation