An In-Depth Introduction to Logistic Regression for Binary Classification

Logistic regression is one of the most popular and widely used machine learning algorithms, especially for binary classification problems. In this blog post, we‘ll dive deep into the inner workings of logistic regression, understand how it differs from linear regression, implement it in Python, and look at best practices for training and evaluating logistic regression models.

What is Logistic Regression?

Logistic regression is a supervised machine learning algorithm used for predicting a binary outcome given a set of independent variables. It models the probability that the outcome belongs to a particular class, typically the positive or "success" class which is labeled 1, with the negative or "failure" class labeled 0.

Some examples of binary classification problems well-suited for logistic regression are:

• Predicting if an email is spam or not spam
• Classifying a tumor as malignant or benign
• Determining if a student will pass or fail an exam
• Forecasting if a customer will churn or not churn

Despite its name, logistic regression is actually a classification algorithm, not a regression algorithm. Regression algorithms predict continuous quantities while classification algorithms predict discrete class labels. The "regression" in logistic regression refers to the fact that it uses a linear regression equation under the hood before transforming the output into a probability with the sigmoid function. We‘ll see how this works in the next section.

Understanding the Logistic Regression Model

At its core, logistic regression is a linear model that makes predictions by computing a weighted sum of the input features plus a bias term, similar to linear regression:

z = w₀ + w₁x₁ + w₂x₂ + … + w_px_p

where:

• z is the logit or log-odds
• w₀ is the bias or intercept term
• w₁ to w_p are the feature weights or coefficients
• x₁ to x_p are the p independent features

However, for binary classification, we don‘t use z directly as the predicted output since it can be any real number, positive or negative. Instead, we squash z using the sigmoid function to get predicted probabilities between 0 and 1:

σ(z) = 1 / (1 + e^-z)

The sigmoid function has an S-shaped curve that asymptotically approaches 0 for large negative values of z and approaches 1 for large positive values of z.

Putting it together, the logistic regression model computes the probability that the outcome belongs to class 1 as:

P(y=1 | x) = 1 / (1 + exp(-w₀ – w₁x₁ – w₂x₂ – … – w_px_p))

To make predictions, we select a decision threshold, typically 0.5, and classify instances with a predicted probability greater than the threshold as class 1, and instances with a predicted probability less than the threshold as class 0. The weights w determine the decision boundary between the classes.

Logistic Regression vs Linear Regression

Why can‘t we just use linear regression for binary classification problems? There are a couple key reasons:

1. Linear regression predictions are unbounded and can take values outside the range [0, 1], which is invalid for probabilities. Squashing the output of linear regression with the sigmoid function constrains it to a valid probability range.

2. With linear regression, the cost function is the residual sum of squares between the predicted and actual values. But for classification, what we really care about is the likelihood of the observed class labels given the data. Logistic regression directly maximizes this likelihood during training.

In logistic regression, the model parameters are chosen to maximize the likelihood of the training data. This is known as maximum likelihood estimation (MLE). The likelihood can be interpreted as the probability of the observed data under the model‘s predictions. Maximizing the likelihood is equivalent to minimizing the negative log-likelihood, so in practice we minimize the negative log-likelihood as the cost function during training, typically with gradient descent.

Unlike linear regression which has a closed-form solution for the optimal parameters using ordinary least squares, logistic regression must be solved numerically with an iterative optimization algorithm since the negative log-likelihood is non-convex. Popular optimization algorithms for logistic regression include gradient descent, Newton‘s method, and L-BFGS.

Evaluating Logistic Regression Models

Once we‘ve trained a logistic regression model, how can we measure its performance and know if it‘s any good? Since logistic regression is a classification model, we need to use classification-specific evaluation metrics rather than regression metrics like mean squared error.

The simplest metric is accuracy, which measures the percentage of examples that were correctly classified by the model. However, accuracy can be misleading for imbalanced datasets where one class is much more frequent than the other. In this case, the model can achieve high accuracy by simply predicting the majority class for every example.

Other common evaluation metrics that give a more complete picture of classification performance are:

• Precision: Of the examples predicted as positive, what fraction are actually positive? Also known as the positive predictive value.
• Recall: Of the actual positive examples, what fraction did the model correctly identify as positive? Also known as sensitivity or the true positive rate.
• F1 score: The harmonic mean of precision and recall. An F1 score reaches its best value at 1 and worst value at 0.
• ROC AUC: The area under the receiver operating characteristic curve plots the true positive rate against the false positive rate as the decision threshold is varied. An AUC of 1 represents a perfect classifier and an AUC of 0.5 represents a classifier that performs no better than random guessing.
• Log loss: Measures the uncertainty of the model‘s predicted probabilities. A model that is very confident in its correct predictions will have a low log loss, while a model that is very confident in its incorrect predictions will have a high log loss.

It‘s important to consider multiple evaluation metrics when assessing classification models to get a holistic view of their strengths and weaknesses.

Implementing Logistic Regression in Python

Now that we understand the theory behind logistic regression, let‘s see how to implement it in Python with the popular scikit-learn library. We‘ll use the Titanic dataset from Kaggle to predict passenger survival.

First, we‘ll load the data, split it into features and target, and divide into train and test sets:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv(‘titanic.csv‘)

features = [‘Pclass‘, ‘Sex‘, ‘Age‘, ‘SibSp‘, ‘Parch‘, ‘Fare‘, ‘Embarked‘]
X = df[features]
X = pd.get_dummies(X) # one-hot encode categorical features
y = df[‘Survived‘]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Next, we create a logistic regression model instance, fit it to the training data, and evaluate it on the test data:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score, log_loss

model = LogisticRegression(random_state=42)
model.fit(X_train, y_train)

y_pred_proba = model.predict_proba(X_test)[:,1]
y_pred = model.predict(X_test)

print(f‘Accuracy: {accuracy_score(y_test, y_pred):.3f}‘)  
print(f‘Precision: {precision_score(y_test, y_pred):.3f}‘)
print(f‘Recall: {recall_score(y_test, y_pred):.3f}‘)    
print(f‘F1 score: {f1_score(y_test, y_pred):.3f}‘)
print(f‘ROC AUC: {roc_auc_score(y_test, y_pred_proba):.3f}‘)  
print(f‘Log loss: {log_loss(y_test, y_pred_proba):.3f}‘)

This will output the various evaluation metrics on the test set:

Accuracy: 0.799
Precision: 0.778
Recall: 0.679
F1 score: 0.725  
ROC AUC: 0.832
Log loss: 0.468

Our logistic regression model achieved an accuracy of about 80% and an ROC AUC of 0.83 on the test set, which is quite good for this problem. The other metrics like precision, recall, and F1 score could potentially be improved if we tune the decision threshold.

Extensions and Best Practices

We‘ve covered binary logistic regression in depth, but there are a couple noteworthy extensions:

1. Multinomial logistic regression is used when the response variable has three or more unordered classes. It‘s also known as softmax regression and uses the softmax function to squash the outputs to probabilities that sum to 1 across all classes.

2. Ordinal logistic regression is used when the response variable has three or more ordered classes. It uses a proportional odds model to estimate the probability of each class.

Some best practices to keep in mind when working with logistic regression are:

• Scale your features to a similar range (e.g. with standardization or normalization) since the model is sensitive to feature scale.
• Address class imbalance in the training data with techniques like upsampling the minority class, downsampling the majority class, or using class weights.
• Be mindful of multicollinearity in your features which can make the model unstable. Remove highly correlated features if necessary.
• Consider using regularization techniques like L1 or L2 regularization if you have a large number of features to prevent overfitting and improve generalization. The scikit-learn LogisticRegression class provides built-in parameters for this.
• Don‘t forget to explore your data with visualizations and summary statistics before diving into modeling!

The main advantages of logistic regression are that it‘s a simple, interpretable model that trains quickly and works well on linearly separable classes. It also provides well-calibrated predicted probabilities that can be useful for decision-making. The main limitation is that it can‘t learn complex non-linear decision boundaries, so more sophisticated algorithms like neural networks may be necessary for challenging classification tasks.

Additional Resources

To learn more about logistic regression, check out these resources:

• StatQuest‘s excellent video: Logistic Regression Details Pt1: Coefficients
• Andrew Ng‘s machine learning course on Coursera: Logistic Regression
• Scikit-learn User Guide: Logistic regression
• Book: An Introduction to Statistical Learning

I hope this in-depth introduction to logistic regression for binary classification has been helpful! Let me know if you have any other questions.