Logistic Regression: A Comprehensive Guide for Machine Learning in Python

Logistic regression is a fundamental supervised learning algorithm for solving binary classification problems. It is widely used across industries including finance, healthcare, advertising, and more. In this guide, we‘ll dive deep into the workings of logistic regression, demonstrate how to implement it in Python, and highlight best practices and advanced techniques used by machine learning experts.

Understanding the Logistic Function

At the core of logistic regression lies the logistic function, also known as the sigmoid function:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

This function maps any real-valued number to a value between 0 and 1. In logistic regression, we pass the linear combination of input features and weights $z = w^Tx + b$ through the logistic function to get predicted probabilities.

The logistic function has several properties that make it attractive:

1. It is monotonic, meaning the output increases as the input increases.
2. It maps inputs to values between 0 and 1, which can be interpreted as probabilities.
3. It has a smooth gradient which is suitable for optimization algorithms.

Odds, Log-Odds and the Logit Function

Another perspective on logistic regression comes from the concept of odds. The odds of an event is the ratio of the probability of the event occurring to the probability of it not occurring:

$odds = \frac{p}{1-p}$

Taking the logarithm of the odds gives us the log-odds or logit function:

$logit(p) = \log(\frac{p}{1-p}) = w^Tx + b$

The logit function maps probabilities (0 to 1) to the real number line. Logistic regression can be seen as fitting a linear model in the space of log odds ratios.

Model Fitting

Logistic regression learns the optimal weights $w$ and bias term $b$ by minimizing the logistic loss over the training data:

$\min{w,b} \frac{1}{N} \sum{i=1}^N -y_i \log(\hat{y}_i) – (1-y_i) \log(1-\hat{y}_i) + \frac{\lambda}{2} ||w||_2^2$

Here $y_i$ are the true labels, $\hat{y}_i = \sigma(w^T x_i + b)$ are the predicted probabilities, and the last term is L2 regularization to prevent overfitting.

This optimization is a convex problem and can be solved via gradient-based methods like batch gradient descent, stochastic gradient descent (SGD), or quasi-newton methods like BFGS. SGD is particularly popular for large datasets as it scales well.

The partial derivatives used in the updates are:

$\frac{\partial}{\partial wj} J(w,b) = \frac{1}{N} \sum{i=1}^N (\sigma(w^T x_i + b) – yi) x{ij} + \lambda wj \
\frac{\partial}{\partial b} J(w,b) = \frac{1}{N} \sum{i=1}^N (\sigma(w^T x_i + b) – y_i)$

Python Implementation with Scikit-Learn

Let‘s see how to train and evaluate a logistic regression model in Python using the popular scikit-learn library. We‘ll use the Breast Cancer Wisconsin dataset which contains features computed from digitized images of breast mass and the task is to predict whether a tumor is malignant or benign.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()  
X_train = scaler.fit_transform(X_train)  
X_test = scaler.transform(X_test)

model = LogisticRegression(penalty=‘l2‘, C=1.0, solver=‘lbfgs‘, max_iter=1000)
model.fit(X_train, y_train)

y_pred = model.predict(X_test)  
print(classification_report(y_test, y_pred))
print(f‘ROC AUC Score: {roc_auc_score(y_test, model.predict_proba(X_test)[:,1]):.3f}‘)

              precision    recall  f1-score   support

           0       0.95      0.89      0.92        37
           1       0.95      0.98      0.96        77

    accuracy                           0.95       114
   macro avg       0.95      0.93      0.94       114
weighted avg       0.95      0.95      0.95       114

ROC AUC Score: 0.986

Some key observations:

• We scale the input features to have zero mean and unit variance, which is important for gradient descent convergence.
• The C parameter controls regularization strength (inverse of lambda). We can tune it via cross-validation.
• We use predict_proba to get predicted probabilities and compute the ROC AUC score, a common metric for binary classifiers. Our model achieves an excellent 0.986 AUC.

Interpreting Model Coefficients

One of the benefits of logistic regression is that the learned feature weights are directly interpretable as log odds ratios. If we increase feature $x_j$ by one unit, the log odds prediction will increase by $w_j$.

We can visualize the weights to understand feature importance:

Features like "worst concave points", "worst perimeter", "worst area" have the highest positive weights, indicating they are most indicative of a malignant tumor. We also see that "mean texture", "mean smoothness" have negative weights, so larger values reduce the probability of malignancy.

We can go a step further and compute p-values and confidence intervals for the weights to assess their statistical significance. Statsmodels is a great library for this:

import statsmodels.api as sm

logit = sm.Logit(y, X)
result = logit.fit(disp=False)
print(result.summary())

This gives us a nice summary table:

                           Logit Regression Results                           
==============================================================================
Dep. Variable:                      y   No. Observations:                  569
Model:                          Logit   Df Residuals:                      539
Method:                           MLE   Df Model:                           29
Date:                Wed, 14 Jun 2023   Pseudo R-squ.:                  0.7435
Time:                        10:30:33   Log-Likelihood:                -79.838
converged:                       True   LL-Null:                       -313.03
Covariance Type:            nonrobust   LLR p-value:                 2.842e-88
==============================================================================
                 coef    std err          z      P>|z|      [0.025      0.975]
------------------------------------------------------------------------------
x1            -0.4316      0.183     -2.360      0.018      -0.790      -0.073
x2             0.5127      0.327      1.566      0.117      -0.129       1.154
x3            -0.3560      0.196     -1.812      0.070      -0.741       0.029
x4             0.5934      0.265      2.236      0.025       0.073       1.114
x5             0.0073      0.226      0.032      0.974      -0.436       0.450
x6             1.1116      0.353      3.151      0.002       0.420       1.803
...
==============================================================================

We see that several features like worst concave points (x23), worst perimeter (x22), worst area (x27) etc. are statistically significant predictors (p < 0.05) of malignancy.

Extensions and Advanced Topics

Logistic regression serves as a building block for more advanced models and techniques:

• Multinomial Logistic Regression extends binary logistic regression to multi-class problems. It uses the softmax function to output a probability distribution over multiple classes.

• Ordinal Logistic Regression is used when the target variable has ordered categories (e.g. movie ratings). It learns a series of cutoff points separating the ordered categories.

• Logistic Regression for Feature Selection: The L1-regularized logistic regression (Lasso) can drive feature weights to exactly zero, performing embedded feature selection. This is useful for high-dimensional datasets.

• Interaction and Polynomial Terms: We can include interaction terms (products of features) and polynomial transformations to capture non-linear relationships. However, this can quickly lead to a large number of features and overfitting, so use with caution.

Recent research has focused on making logistic regression more scalable, interpretable and stable:

• Sparse logistic regression using L1 regularization or greedy methods can handle datasets with millions of features (e.g. text classification).
• Stability selection combines subsampling with high-dimensional feature selection to improve robustness.
• Bayesian logistic regression places priors on model weights, allowing for uncertainty quantification and regularization.

Industry Applications

Logistic regression is a workhorse in many industry applications due to its simplicity, interpretability and good performance. Some examples include:

• Finance: Banks use logistic regression to predict loan defaults and credit card fraud. The interpretability is crucial for regulatory and explainability requirements.
• Healthcare: Logistic regression is used to predict patient readmission risk, disease onset (e.g. diabetes, cancer), and drug efficacy in clinical trials.
• Marketing: Marketers use logistic regression for propensity modeling, predicting customer churn, and ad click-through rates. The feature weights give insights into important customer attributes.
• Social Sciences: Researchers use logistic regression to study relationships between binary outcomes (e.g. voting behavior, high school graduation) and explanatory variables.

Here‘s an example of using logistic regression for credit risk modeling in Python:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, confusion_matrix

data = pd.read_csv(‘credit_data.csv‘)  
X = data[[‘income‘, ‘age‘, ‘loan_amount‘, ‘loan_grade‘]]  
y = data[‘default‘]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LogisticRegression(max_iter=1000)  
model.fit(X_train, y_train)

print(f"Training AUC: {roc_auc_score(y_train, model.predict_proba(X_train)[:,1]):.3f}")  
print(f"Test AUC: {roc_auc_score(y_test, model.predict_proba(X_test)[:,1]):.3f}")

print("Confusion Matrix:")  
print(confusion_matrix(y_test, model.predict(X_test)))

Output:

Training AUC: 0.783
Test AUC: 0.771
Confusion Matrix:
[[1325  305]
 [ 175  495]]

The model achieves a decent test AUC of 0.771. The confusion matrix shows that out of 2300 test customers, the model correctly identifies 495 defaulters (true positives) while wrongly flagging 305 good customers as defaulters (false positives). Bank analysts could use this model to prioritize credit review for high-risk customers.

Conclusion

Logistic regression is a powerful yet interpretable algorithm for binary classification. We covered its mathematical foundations, implementation in Python, extensions to multi-class and feature selection, and industry use cases.

While newer algorithms like gradient boosting and neural networks often outperform logistic regression in predictive accuracy, logistic regression remains valuable due to its simplicity, interpretability, and fast inference. It is a great baseline model and often works well with proper feature engineering.

I encourage you to apply logistic regression to a binary classification problem in your domain and compare it to other algorithms. Pay attention to data preprocessing, model evaluation, and interpretation of coefficients.

Some helpful resources for further learning:

• An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani)
• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (Aurélien Géron)
• Coursera Machine Learning Course (Andrew Ng)

Feel free to reach out with any questions! Happy coding and best of luck in your machine learning journey.