A Deep Dive into Logistic Regression: Mastering the Logit Function in R

Introduction

Logistic regression is one of the most widely used statistical models for binary classification problems. Despite its popularity and inclusion in many statistical software packages, it‘s valuable for data scientists to understand the mechanics of how logistic regression works under the hood.

In this post, we‘ll take a deep dive into the fundamentals of logistic regression, with a focus on the core component – the logit function. We‘ll derive the logistic equation from scratch and build our own logistic regression model in R to cement our understanding. Let‘s dive in!

What is Logistic Regression?

Logistic regression is a classification algorithm used to assign observations to a discrete set of classes. Unlike linear regression which outputs continuous number values, logistic regression transforms its output using the logistic sigmoid function to return a probability value which can then be mapped to two or more discrete classes.

Logistic regression is used widely across many domains from healthcare to marketing. Some common use cases include:

• Predicting risk of a disease or condition (e.g. risk of cancer based on patient characteristics)
• Classifying email as spam or not spam
• Estimating likelihood of a customer purchasing a product
• Forecasting probability of equipment failure

The core concept that underlies logistic regression is the logit function. Let‘s take a closer look at this function and understand how it powers logistic regression under the hood.

The Logit Function

At the heart of logistic regression is the logit function. The logit function is defined as the logged odds:

logit(p) = log(p/(1-p))

where p is the probability of the desired outcome. The logit function takes a probability value between 0 and 1 and maps it to a real number value between negative and positive infinity. It is the inverse of the sigmoid function which maps real number values back to a probability between 0 and 1.

Some important properties of the logit function:

• It is monotonically increasing – as p increases, so does logit(p)
• When p=0.5, logit(p)=0
• As p approaches 0, logit(p) approaches negative infinity
• As p approaches 1, logit(p) approaches positive infinity

The logit is the natural log of the odds, where odds are defined as the ratio of the probability of success to the probability of failure:

odds = p/(1-p)

Positive logit values mean the odds are greater than 1 (p > 0.5) and the event is more likely to happen than not. Negative logit values mean the odds are less than 1 (p < 0.5) and the event is less likely.

Logistic regression takes a linear combination of the predictors, and models the logit (log-odds) as a linear function:

logit(p) = log(p/(1-p)) = β0 + β1X1 + β2X2 + … + βnXn

The logistic equation can then be rearranged to solve for the probability p:

p = exp(β0 + β1X1 + β2X2 + … + βnXn) / 1+exp(β0 + β1X1 + β2X2 + … + βnXn)

This is the fundamental equation that powers logistic regression. The model coefficients (the βs) are learned from training data using maximum likelihood estimation.

Building a Logistic Regression Model from Scratch in R

To fully understand the mechanics of logistic regression, there‘s no better way than implementing it from scratch. We‘ll build our own logistic regression model in R without using any existing packages.

Let‘s consider a toy dataset of 10 observations:

X = {1,2,3,4,5,6,7,8,9,10}

Y = {0,0,0,0,1,0,1,0,1,1}

Setting up the data

First, let‘s create our dataset in R:

x <- 1:10
y <- c(0,0,0,0,1,0,1,0,1,1)
data <- data.frame(x=x, y=y)

Deriving the logistic regression equation

Next, let‘s derive our logistic regression equation. Recall that logistic regression models the log-odds (logit) of the desired outcome as a linear combination of the predictors:

logit(p) = β0 + β1*X

We can rearrange this to solve for the probability p:

p = exp(β0 + β1X) / (1 + exp(β0 + β1X))

Likelihood function

To find the optimal values for the model coefficients β0 and β1, we use maximum likelihood estimation. The likelihood function L(β) for logistic regression is:

L(β) = Π p(Xi)^yi * (1-p(Xi))^(1-yi)

where:

• Π denotes the product over all data points
• p(Xi) is the predicted probability for the ith data point
• yi is the actual class for the ith data point

It‘s easier to work with the log of the likelihood function, called the log-likelihood:

LL(β) = Σ yilog(p(Xi)) + (1-yi)log(1-p(Xi))

Our goal is to find the values of β that maximize the log-likelihood.

Optimizing parameters with Newton-Raphson

To find the optimal β parameters, we‘ll use the Newton-Raphson optimization method. This is an iterative method that uses the first and second derivatives of the log-likelihood function to update the estimates of β at each step.

The Newton-Raphson update rule is:

βnew = βold – H^(-1)*g

where:

• H is the Hessian matrix of second-order partial derivatives of LL(β)
• g is the gradient vector of first-order partial derivatives of LL(β)

Here is the R code to implement logistic regression with the Newton-Raphson method:

logistic <- function(x, y, num_iter=100) {
  x <- cbind(1, x)  # add intercept column
  n <- nrow(x)

  beta <- rep(0, ncol(x))  # initialize betas to 0

  for (i in 1:num_iter) {
    p <- 1 / (1 + exp(-x %*% beta))
    W <- diag(as.vector(p * (1-p)))

    grad <- t(x) %*% (y - p)    
    hess <- -t(x) %*% W %*% x

    beta <- beta - solve(hess, grad)
  }

  return(beta)
}

Let‘s run our logistic function on the toy dataset:

logistic(x, y)

            x 
-4.077357 1.540047 

The final β values are β0=-4.08 and β1=1.54.

Comparing to glm() results

Now let‘s compare our results to the output from the glm() function in R:

glm(y ~ x, data=data, family="binomial")

Coefficients:
(Intercept)            x  
  -4.077358    1.540047

We can see that our hand-coded logistic regression function produces identical results to R‘s built-in glm() function.

Interpreting the Model

Now that we have fit our logistic regression model, how do we interpret the coefficients?

The β coefficients in logistic regression represent the change in the log-odds of the outcome for a one unit increase in the predictor variable, holding other variables constant.

Since the logistic model is fit on the log-odds scale, the coefficients are interpreted as additive effects on the log-odds scale. To convert to a multiplicative effect on the odds scale, we can exponentiate the coefficients.

In our example:

• β1 = 1.54 means that for every one unit increase in X, the log-odds of the outcome increases by 1.54
• exp(β1) = exp(1.54) = 4.67 means that for every one unit increase in X, the odds of the outcome increases by a factor of 4.67

We can also use the model to make probability predictions on new data:

beta <- logistic(x, y)
x_new <- data.frame(x=seq(1,10,0.1))
p <- 1 / (1 + exp(-cbind(1, x_new) %*% beta))

plot(data$x, data$y)
lines(x_new$x, p, col=‘blue‘)

This plots the original data points along with the logistic curve representing the predicted probabilities.

Evaluating Model Performance

To evaluate the performance of a logistic regression model, we can use a variety of metrics depending on the use case.

Some common evaluation metrics for binary classification:

• Accuracy: (TP+TN) / (TP+TN+FP+FN)
• Precision: TP / (TP+FP)
• Recall: TP / (TP+FN)
• F1-score: 2PrecisionRecall / (Precision+Recall)
• ROC AUC: Area under the ROC curve

Where:

• TP = True Positives
• TN = True Negatives
• FP = False Positives
• FN = False Negatives

We can calculate these metrics by comparing the actual values to the predicted probabilities at different classification thresholds.

Another useful tool is the ROC (Receiver Operating Characteristic) curve, which plots the true positive rate vs the false positive rate at different classification thresholds. The area under the ROC curve, or AUC, is a useful aggregate measure of classifier performance. An AUC of 0.5 indicates a classifier that performs no better than random guessing, while an AUC of 1 indicates a perfect classifier.

Extensions and Alternatives

While the vanilla logistic regression model is very useful, there are a number of extensions and alternative models to be aware of:

• Multinomial logistic regression extends logistic regression to problems where the dependent variable can take more than two classes.

• Ordinal logistic regression handles ordered dependent variables (e.g. movie ratings on a 1-5 scale).

• Regularized logistic regression (using techniques like lasso or ridge) can help prevent overfitting and deal with multicollinearity.

• Generalized linear models extend logistic regression to other types of dependent variables like counts or continuous variables.

• Discriminant analysis is a related classification technique that assumes the predictors are normally distributed.

• Tree-based algorithms like decision trees and random forests offer a non-parametric alternative to logistic regression.

• Neural network models can capture more complex non-linear relationships between the predictors and the outcome.

Conclusion

Logistic regression is a powerful and widely used model for binary classification. The logit function is the key component that allows logistic regression to model probabilities and classes.

By implementing logistic regression from scratch in R, we gain a deeper understanding of the underlying mechanics of how the model works. We‘ve seen how to derive the logistic equation, use maximum likelihood estimation to learn the optimal model coefficients, and evaluate model performance on real data.

While modern machine learning software makes it easy to fit a logistic regression with a single function call, understanding the fundamentals is invaluable for aspiring data scientists. I hope this deep dive has helped solidify your understanding of logistic regression and given you the confidence to implement it yourself from scratch.