Beyond Accuracy: A Deep Dive into Log Loss for Logistic Regression

Logistic regression is a fundamental model in the machine learning toolkit, particularly well-suited for binary classification problems. While accuracy is often the go-to metric for evaluating classifiers, it can be misleading and hide important nuances in performance. In this blog post, we will take a deep dive into log loss, also known as binary cross entropy – the de facto standard loss function for training and evaluating logistic regression models. Far from a simple implementation detail, log loss has deep connections to information theory and probability calibration that make it a theoretically principled and practically useful metric to understand.

Mathematical Formulation

First, let‘s establish some notation. We have a labeled dataset of $n$ examples ${(x_i, yi)}{i=1}^n$, where $x_i$ is an input feature vector and $y_i \in {0,1}$ is the corresponding binary label. Our goal is to learn a logistic regression model parameterized by weights $w$ and bias term $b$ that outputs the probability $p_i$ of the positive class for each input $x_i$:

$$p_i = P(y_i = 1 | x_i) = \sigma(w^T x_i + b)$$

where $\sigma(z) = \frac{1}{1 + e^{-z}}$ is the sigmoid function that squashes the linear combination $w^T x_i + b$ to be between 0 and 1.

The log loss for this model over the entire dataset is defined as:

$$\mathcal{L}(w, b) = -\frac{1}{n} \sum_{i=1}^n [y_i \log p_i + (1-y_i) \log (1-p_i)]$$

This formula might look intimidating, but it has an elegant interpretation. The term $y_i \log p_i$ is the log probability of the true label, which equals $\log p_i$ if $y_i=1$ and $0$ if $y_i=0$. Similarly, the term $(1-y_i) \log (1-p_i)$ is the log probability of the incorrect label.

Putting these together, log loss is simply the negative average log probability of the true label across all examples. Minimizing log loss means maximizing the log probability of the data under the model, which is exactly the objective in maximum likelihood estimation! Mathematically, we have:

$$\begin{aligned}
\mathcal{L}(w, b) &= -\frac{1}{n} \sum_{i=1}^n \log P(y_i|xi) \
&= -\frac{1}{n} \log \prod{i=1}^n P(y_i|x_i) \
&= -\frac{1}{n} \log P({yi}{i=1}^n | {xi}{i=1}^n)
\end{aligned}$$

The first line is the definition of log loss, the second line follows from the independence assumption between examples, and the third line is the log likelihood of the dataset under the model. Thus, minimizing log loss is equivalent to maximum likelihood estimation for logistic regression.

A Better Metric than Accuracy

Consider a logistic regression model that predicts $p_1 = 0.6$ and $p_2 = 0.8$ for two positive examples $(x_1, y_1=1)$ and $(x_2, y_2=1)$. With a decision threshold of 0.5, both will be classified correctly and the model will have 100% accuracy. However, are these predictions equally good? The log loss tells a different story:

$$\begin{aligned}
\mathcal{L}_1 &= -\log 0.6 = 0.51 \
\mathcal{L}_2 &= -\log 0.8 = 0.22
\end{aligned}$$

The second prediction has lower log loss and is intuitively better since it is more confident in the correct class. An accuracy of 100% can hide these differences in prediction quality.

Log loss is a proper scoring rule, which means it is minimized by the true underlying probability distribution. It penalizes confident predictions that are wrong more heavily than uncertain predictions that happen to be right. This encourages models to be well-calibrated and output probabilities that accurately reflect the true likelihood of each class.

To illustrate, let‘s compare the log loss of three models that all have 95% accuracy on a balanced test set with 1000 positive and 1000 negative examples:

Model A‘s predictions are well-calibrated, Model B is overconfident, and Model C is underconfident. Log loss correctly ranks B > A > C, while accuracy cannot distinguish between them. Choosing the decision threshold to maximize accuracy does not guarantee the best log loss.

Connection to Information Theory

Log loss has a fascinating connection to fundamental concepts in information theory, which studies the transmission and compression of data. Cross entropy, a generalization of log loss for multi-class problems, measures the average number of bits needed to encode a message from a true distribution $p$ using a code optimized for a predicted distribution $q$:

$$H(p,q) = \mathbb{E}p[-\log q] = -\sum{i=1}^n p_i \log q_i$$

Minimizing cross entropy is equivalent to minimizing the Kullback-Leibler (KL) divergence $D_{KL}(p||q)$ between $p$ and $q$:

$$D{KL}(p||q) = \sum{i=1}^n p_i \log \frac{p_i}{q_i} = H(p,q) – H(p)$$

where $H(p) = -\sum_{i=1}^n p_i \log p_i$ is the entropy of $p$. KL divergence measures the information lost when approximating $p$ with $q$, and is minimized when $q=p$. Thus, log loss is minimized when the predicted probabilities match the true data distribution, a stronger condition than simply maximizing accuracy!

Calibration is Key

For log loss to be meaningful, it is crucial that the predicted probabilities are well-calibrated – i.e. they accurately represent the true likelihood of each class. A model that outputs 0.8 for every positive example and 0.2 for every negative example will have excellent accuracy and log loss, but its predictions are meaningless as actual probabilities.

Assessing calibration is typically done by bucketing examples into bins based on predicted probability, then comparing the average predicted probability to the empirical fraction of positive examples in each bin. A perfectly calibrated model will have these two quantities match up across all bins.

There are several techniques to improve calibration, such as Platt scaling and isotonic regression, which learn a monotonic transformation of the model‘s scores to better match the empirical probabilities. Using a held-out validation set for calibration is essential to avoid overfitting.

Practical Tips

Here are some best practices to keep in mind when using log loss for logistic regression:

1. Scale your features: Log loss is sensitive to the scale of the inputs, so standardizing or normalizing features can improve convergence speed and stability during training.

2. Handle class imbalance: When the classes are imbalanced, a model that always predicts the majority class can achieve low log loss. Upsampling the minority class, downsampling the majority class, or adjusting the decision threshold are common strategies to combat this.

3. Tune the regularization: Adding L1 or L2 penalties on the model weights can help prevent overfitting and improve generalization. Use cross-validation to select the best regularization strength.

4. Monitor validation performance: Track log loss on a validation set during training to detect overfitting and select the best model checkpoint. Be wary of models with decreasing training loss but increasing validation loss.

5. Understand your domain: The same log loss value can mean different things for different problems. A log loss of 0.2 might be excellent for a challenging problem like medical diagnosis but terrible for a simple problem like spam detection. Always compare to a meaningful baseline.

Beyond Binary Classification

While we focused on binary classification in this post, log loss easily extends to multi-class problems via the softmax function, which generalizes the sigmoid to $K$ classes:

$$P(y=k|x) = \frac{e^{wk^T x}}{\sum{j=1}^K e^{w_j^T x}}$$

The log loss for a multi-class problem with $K$ classes is:

$$\mathcal{L}(w) = -\frac{1}{n} \sum{i=1}^n \sum{k=1}^K y{ik} \log p{ik}$$

where $y{ik}$ is 1 if example $i$ belongs to class $k$ and 0 otherwise, and $p{ik}$ is the predicted probability of class $k$ for example $i$.

Logistic regression and log loss can even be used for multi-label problems, where each example can belong to multiple classes simultaneously, by training a separate binary classifier for each class. More generally, structured prediction problems with exponentially many possible outputs (e.g. sequences, trees) often use a variant of log loss called conditional log-likelihood.

Current Research

Log loss and logistic regression are still active areas of research, with recent work focusing on:

• Non-convex losses: Log loss is convex, but non-convex alternatives like focal loss and F-measure loss have been proposed to handle class imbalance and noisy labels.

• High-dimensional data: When the number of features is much larger than the number of examples, classical logistic regression can overfit. Regularized versions like L1-penalized logistic regression can learn sparse models in this setting.

• Online learning: Stochastic gradient descent allows logistic regression to be updated incrementally as new data arrives, making it well-suited for online learning settings.

• Transfer learning: Pre-trained language models like BERT can be fine-tuned for downstream tasks by adding a logistic regression layer on top and optimizing log loss.

• Uncertainty quantification: Bayesian logistic regression places a prior distribution on the model weights and can estimate the uncertainty of its predictions.

Real-World Applications

Logistic regression and log loss are ubiquitous in industry, with applications including:

• Click-through rate prediction: Online advertising systems use logistic regression to predict the probability that a user will click on an ad.

• Fraud detection: Banks and credit card companies use logistic regression to flag potentially fraudulent transactions based on user behavior and transaction history.

• Disease diagnosis: Medical researchers use logistic regression to predict the likelihood of a patient having a disease based on their symptoms and risk factors.

• Churn prediction: Subscription-based businesses use logistic regression to predict which customers are likely to churn and target them with retention offers.

Conclusion

We‘ve covered a lot of ground in this deep dive into log loss for logistic regression, from its mathematical formulation and theoretical properties to practical tips and real-world applications. To recap:

• Log loss is the negative average log likelihood of the true labels, and minimizing it is equivalent to maximum likelihood estimation.
• Log loss is a better metric than accuracy for evaluating the quality of predicted probabilities, and is minimized when the predictions match the true data distribution.
• Log loss has deep connections to information theory concepts like cross entropy and KL divergence.
• To use log loss effectively, it‘s important to scale features, handle class imbalance, tune regularization, monitor validation performance, and understand the problem domain.
• Logistic regression and log loss extend beyond binary classification to multi-class, multi-label, and structured prediction problems.
• Current research on log loss focuses on non-convex losses, high-dimensional data, online learning, transfer learning, and uncertainty quantification.
• Log loss is widely used in industry for applications like click-through rate prediction, fraud detection, disease diagnosis, and churn prediction.

We hope this post has given you a deeper appreciation for the elegance and power of log loss as a tool for training and evaluating logistic regression models. While it may seem like a simple concept, log loss is a cornerstone of machine learning theory and practice that every data scientist should have in their toolkit. Thanks for reading!