# Logistic Regression and Maximum Likelihood Explained Simply

- Canonical: https://33rdsquare.com/logistic-regression-and-maximum-likelihood-explained-simply-part-i/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

In a previous article, we looked at linear regression and saw how it could be used to model the relationship between a continuous input variable and a continuous output variable. Linear regression assumes a linear relationship between the inputs and output, and finds the linear model that minimizes the mean squared error between the model‘s predictions and the true outputs.

However, linear regression is not suitable for all prediction problems. One major limitation is that it cannot directly model classification problems where the output is a discrete class label rather than a continuous value. For example, classifying an email as spam or not spam based on its text content, or diagnosing a patient as having a disease or not based on their symptoms and test results. In these binary classification problems, we need a model that can predict a probability between 0 and 1 of the input belonging to the positive class. This is where logistic regression comes in.

## From Linear to Logistic Regression

Logistic regression gets its name from the logistic function, also known as the sigmoid function, which forms the core of the model. The logistic function maps any real-valued number to a value between 0 and 1. Mathematically, it‘s defined as:

σ(z) = 1 / (1 + e^-z)

Where z is the input and e is the mathematical constant approximately equal to 2.71828. If we plot the logistic function, we see that it has an S-shaped curve:

[Image of logistic/sigmoid curve]

The key property of the logistic function is that it can take any real-valued number and map it to a value between 0 and 1, which can be interpreted as a probability. Very negative values of z get mapped to values close to 0, while very positive values of z approach 1.

In logistic regression, we calculate z as a linear combination of the input features and model parameters, just like in linear regression:

z = w0 + w1_x1 + w2_x2 + … + wn*xn

The weights w0, w1, etc. are the model parameters that we need to learn from the training data. However, unlike linear regression, we don‘t use z directly as the model output. Instead, we pass it through the logistic function to get a probability between 0 and 1:

p(y=1|x) = σ(z)
 = 1 / (1 + e^-(w0 + w1_x1 + w2_x2 + … + wn*xn))

Here p(y=1|x) represents the conditional probability that the true class label y is equal to 1 (the positive class) given the input features x. The model predicts the negative class (y=0) for values of p(y=1|x) less than 0.5 and the positive class for values greater than 0.5.

## Odds and Log-Odds

To understand how logistic regression learns its parameters, we need to talk about two important concepts: odds and log-odds. The odds of an event occurring is defined as the probability of the event divided by the probability of the event not occurring. In logistic regression, the odds of the positive class is given by:

odds(y=1|x) = p(y=1|x) / p(y=0|x)
 = p(y=1|x) / (1 – p(y=1|x))

For example, if the probability of an email being spam is 0.8, then the odds of it being spam is 0.8/0.2 = 4. This means the email is 4 times more likely to be spam than not spam.

The log-odds or logit is simply the logarithm of the odds:

logit(y=1|x) = log(odds(y=1|x))
 = log(p(y=1|x) / (1 – p(y=1|x)))

The logit function maps probabilities in the range (0, 1) to real-valued numbers in (−∞, +∞). The logit function is the inverse of the logistic function, so applying the logistic function to a logit value gives us back the original probability.

Importantly, the logit is equal to the linear combination of inputs and weights that we called z earlier:

z = w0 + w1_x1 + w2_x2 + … + wn*xn
 = logit(y=1|x)

So in logistic regression, we‘re actually fitting a linear model to the log-odds of the positive class. The logistic function then converts the log-odds back to an estimated probability.

## Maximum Likelihood Estimation

Now that we‘ve defined our model, how do we find the optimal values of the parameters w to fit the training data? In linear regression we used ordinary least squares to minimize the mean squared error. In logistic regression, the most common approach is maximum likelihood estimation (MLE).

The basic idea behind MLE is to find the model parameters that maximize the likelihood of observing the training data, assuming the data was actually generated by the model. The likelihood is the probability of the observed data under the model with given parameter values.

For logistic regression on binary classification data with features matrix X and binary class labels vector y, the likelihood function is:

L(w|X,y) = Π p(y=1|x)^yi * (1-p(y=1|x))^(1-yi)

The symbol Π represents the product over all the training examples i=1 to n. For each example, the likelihood is p(y=1|x) if its true label is 1, and (1-p(y=1|x)) if its true label is 0. Multiplying these together gives the probability of the observed labels under the model.

To make the math easier, we usually maximize the log-likelihood instead, since the logarithm is a monotonic function. The log-likelihood is:

l(w) = Σ yi_log(p(y=1|x)) + (1-yi)_log(1-p(y=1|x))

Maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood (NLL). With some algebra, we can show that minimizing the NLL is the same as minimizing the logistic loss or cross-entropy loss between the true labels and predicted probabilities:

NLL(w) = – Σ yi_log(p(y=1|x)) + (1-yi)_log(1-p(y=1|x))
 = – Σ yi_log(σ(w^T_x)) + (1-yi)_log(1-σ(w^T_x))

Where w^T*x is the dot product between the weights vector and an example‘s feature vector, i.e. the linear combination z. So fitting logistic regression by maximum likelihood estimation is equivalent to minimizing the cross-entropy loss between the labels and predictions. This is a convex optimization problem that can be solved using gradient descent methods.

## Model Evaluation

Once we‘ve trained our logistic regression model, how do we evaluate its performance? Since it outputs probabilities, we can‘t use the same metrics as for linear regression like mean squared error.

Some common evaluation metrics for binary classification are:

- Accuracy: Fraction of examples classified correctly
- Precision: Fraction of positive predictions that are truly positive
- Recall: Fraction of true positives captured by positive predictions
- F1 score: Harmonic mean of precision and recall
- ROC curve: Plots true positive rate vs false positive rate as prediction threshold varies
- AUC: Area under the ROC curve; measures model‘s ranking ability

Cross-entropy loss is also a useful metric, since it more heavily penalizes confident misclassifications. Log loss has a nice information-theoretic interpretation as the amount of bits needed to encode the labels using the model‘s predicted probabilities as an optimal code.

## Extensions and Limitations

We‘ve focused on binary logistic regression so far, but it can be extended to multi-class classification problems as well. In multi-class logistic regression, also known as softmax regression or multinomial logistic regression, the model outputs a separate probability for each class and normalizes them to sum to 1. The negative log-likelihood loss function also generalizes straightforwardly to the multi-class case.

Despite its popularity and success, logistic regression has some important limitations to be aware of:

- It assumes a linear decision boundary between classes, which may not hold in practice. Non-linear feature transformations can make the boundary more flexible.
- It‘s not robust to outliers or misclassified examples, which can have an oversized effect on the learned weights. Regularization techniques like L1/L2 penalty on weights can help.
- It tends to underperform in low-data or high-dimensional regimes. Priors on weights or dimensionality reduction may be necessary.
- The logistic loss doesn‘t optimize accuracy directly and may not be calibrated. Techniques like Platt scaling can improve calibration.

More advanced methods like support vector machines, random forests, gradient boosting, and neural networks can outperform logistic regression on many tasks while relaxing some of its assumptions. However, logistic regression remains a powerful and interpretable baseline and is often a key building block in more complex methods.

## Conclusion

Logistic regression is a fundamental model for binary classification problems. By applying the logistic function to a linear combination of input features, it can estimate the conditional probability of the positive class. The model is typically fit by maximum likelihood estimation, which is equivalent to minimizing the logistic loss between the true labels and predicted probabilities.

While a relatively simple model, logistic regression achieves excellent results on many real-world problems and is widely used in industry. Understanding its core concepts of odds, log-odds, and logistic loss is important for any data scientist or machine learning practitioner. I hope this article helped explain these concepts in an accessible way. In future articles, we‘ll dive deeper into more advanced classification methods that build on the ideas introduced here.

---

Source: [Logistic Regression and Maximum Likelihood Explained Simply](https://33rdsquare.com/logistic-regression-and-maximum-likelihood-explained-simply-part-i/)