Logistic Regression: An Introductory Note

Logistic regression is a fundamental machine learning algorithm for binary classification problems. It is widely used across industries for tasks such as fraud detection, churn prediction, and disease diagnosis. Despite its name, logistic regression is a classification algorithm, not a regression algorithm.

At its core, logistic regression is a linear model that predicts the probability of an instance belonging to a particular class. The model learns a decision boundary that separates the two classes. Mathematically, the logistic regression model is defined as:

$P(Y=1|X) = \frac{1}{1+e^{-(\beta_0 + \beta_1X_1 + … + \beta_pX_p)}}$

where $Y$ is the binary target variable, $X=(X_1,…,X_p)$ are the input features, and $\beta=(\beta_0, \beta_1,…,\beta_p)$ are the model coefficients.

The right-hand side of the equation is the logistic or sigmoid function applied to the linear combination of input features. The sigmoid function squashes the output to a probability value between 0 and 1.

$\sigma(z) = \frac{1}{1+e^{-z}}$

The decision boundary learned by logistic regression is linear. For a single input feature, it is a point on the number line. For two features, it is a straight line. For higher dimensions, it is a hyperplane.

Logistic Regression vs Linear Regression

While logistic regression is used for classification, linear regression is used for regression tasks where the target variable is continuous. The key differences are:

• Linear regression predicts a continuous quantity, logistic regression predicts a probability
• Linear regression uses ordinary least squares as the loss function, logistic regression uses maximum likelihood estimation
• Linear regression outputs can be any real number, logistic regression outputs are between 0 and 1

However, the underlying mechanics are similar. Both models learn a linear combination of input features. The difference is in the output transformation – linear regression uses the identity function (no transformation) while logistic regression uses the sigmoid function.

Assumptions

Logistic regression makes several assumptions about the data:

1. Binary target variable
2. No strongly correlated features (multicollinearity)
3. Linearly separable classes
4. Large sample size (at least 10 instances per feature)
5. Independent observations

Violating these assumptions can lead to poor model performance. It‘s important to check the assumptions before applying logistic regression.

Evaluation Metrics

Accuracy is the most commonly used metric to evaluate classifiers. However, accuracy can be misleading for imbalanced datasets. Other metrics like precision, recall, F1 score, and AUC are more informative.

• Precision measures the proportion of true positives among the instances predicted as positive
• Recall measures the proportion of actual positives that are correctly predicted
• F1 score is the harmonic mean of precision and recall
• AUC measures the ability of the classifier to rank positive instances higher than negative instances

Here are the formulae for the metrics:

$Precision = \frac{TP}{TP+FP}$

$Recall = \frac{TP}{TP+FN}$

$F1 = 2 \cdot \frac{Precision \cdot Recall}{Precision+Recall}$

where TP is true positives, FP is false positives, and FN is false negatives.

Scikit-learn provides functions to compute all these metrics:

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score

print("Accuracy:", accuracy_score(y_test, y_pred))
print("Precision:", precision_score(y_test, y_pred))
print("Recall:", recall_score(y_test, y_pred)) 
print("F1 score:", f1_score(y_test, y_pred))
print("AUC:", roc_auc_score(y_test, y_pred_prob[:,1]))

Another useful tool for evaluating classifiers is the confusion matrix. It shows the number of instances in each of the four categories: true positives, true negatives, false positives, false negatives.

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, y_pred)
print(cm)

Interpreting Coefficients

One advantage of logistic regression over black-box models is interpretability. The model coefficients have a clear interpretation as odds ratios.

The odds of an event is the ratio of the probability of the event occurring to the probability of it not occurring. In logistic regression, the coefficients represent the change in log odds for a one unit change in the input feature, holding other features constant.

$\log\left(\frac{P(Y=1|X)}{1-P(Y=1|X)}\right) = \beta_0 + \beta_1X_1 + … + \beta_pX_p$

Exponentiating both sides, we get:

$\frac{P(Y=1|X)}{1-P(Y=1|X)} = e^{\beta_0 + \beta_1X_1 + … + \beta_pX_p}$

So, increasing $X_i$ by one unit multiplies the odds by $e^{\beta_i}$, holding other features constant. If $\beta_i$ is positive, increasing $X_i$ increases the odds of the positive class. If $\beta_i$ is negative, increasing $X_i$ decreases the odds.

We can extract the model coefficients from the trained logistic regression model in scikit-learn:

coefs = pd.DataFrame(
    data = lr.coef_.T,
    columns = [‘Coefficient‘],
    index = data.feature_names
)
coefs[‘Odds Ratio‘] = np.exp(coefs[‘Coefficient‘])
print(coefs)

Feature Selection

In high-dimensional datasets with many features, it‘s important to select the most relevant features for the model. Including irrelevant or redundant features can lead to overfitting, increased training time, and reduced interpretability.

There are three main approaches to feature selection:

1. Filter methods: Select features based on their statistical properties (e.g. correlation with target, variance)
2. Wrapper methods: Select features based on model performance (e.g. recursive feature elimination)
3. Embedded methods: Perform feature selection during model training (e.g. L1 regularization)

Scikit-learn provides several feature selection classes:

from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.feature_selection import RFE

# Filter method
selector = SelectKBest(f_classif, k=5)
X_new = selector.fit_transform(X, y)

# Wrapper method  
selector = RFE(estimator=LogisticRegression(), n_features_to_select=5)
X_new = selector.fit_transform(X, y)

Regularization

Logistic regression is prone to overfitting, especially with high-dimensional data. Regularization is a technique to prevent overfitting by adding a penalty term to the loss function. The two common regularization methods are L1 (Lasso) and L2 (Ridge).

• L1 regularization adds the absolute values of coefficients to the loss function. It tends to produce sparse models by shrinking some coefficients to exactly zero.
• L2 regularization adds the squared values of coefficients to the loss function. It tends to produce models with small but non-zero coefficients.

Scikit-learn‘s LogisticRegression class supports both L1 and L2 regularization via the penalty parameter. The strength of regularization is controlled by the C parameter, which is the inverse of the regularization strength. Smaller values of C correspond to stronger regularization.

lr = LogisticRegression(penalty=‘l1‘, C=0.1, solver=‘liblinear‘)
lr.fit(X_train, y_train)

Multiclass Logistic Regression

Logistic regression can be extended to multiclass classification problems where the target variable has more than two classes. There are two main approaches:

1. One-vs-Rest (OvR): Train a separate binary classifier for each class, treating that class as positive and the rest as negative. For a new instance, predict the class with the highest probability.
2. Multinomial: Train a single model with multiple coefficients, one for each class. For a new instance, predict the class with the highest probability according to the softmax function.

Scikit-learn‘s LogisticRegression class supports both OvR and multinomial logistic regression via the multi_class parameter. The default is OvR.

lr = LogisticRegression(multi_class=‘multinomial‘, solver=‘lbfgs‘)

Case Studies

Logistic regression has been successfully applied across various domains. Here are some industry case studies:

1. Finance: Logistic regression is used to predict loan defaults, credit card fraud, and customer churn in banks and insurance companies. For example, a bank can use logistic regression to predict the probability of a customer defaulting on a loan based on their credit history, income, and other demographic features.

2. Healthcare: Logistic regression is used to predict disease risk, patient readmission, and mortality. For example, a hospital can use logistic regression to predict the probability of a patient being readmitted within 30 days based on their diagnosis, treatment, and demographic features.

3. Marketing: Logistic regression is used to predict customer purchase behavior, ad click-through rates, and email open rates. For example, an e-commerce company can use logistic regression to predict the probability of a customer making a purchase based on their browsing history, past purchases, and demographic features.

4. Natural Language Processing: Logistic regression is used for sentiment analysis, spam detection, and topic classification of text data. For example, a social media platform can use logistic regression to predict the sentiment (positive, negative, neutral) of user posts based on the text content.

Best Practices

Here are some best practices to keep in mind when applying logistic regression:

1. Preprocess the data: Handle missing values, convert categorical variables to numerical, standardize/normalize features
2. Check assumptions: Verify the assumptions of logistic regression before applying the model
3. Split data: Split the data into train, validation, and test sets for model selection and evaluation
4. Use regularization: Apply L1 or L2 regularization to prevent overfitting, especially with high-dimensional data
5. Tune hyperparameters: Use grid search or random search to find the best hyperparameters (e.g. regularization strength)
6. Evaluate with multiple metrics: Use metrics like precision, recall, F1, and AUC in addition to accuracy
7. Interpret the coefficients: Examine the model coefficients to gain insights into the importance and direction of impact of features
8. Use understandable features: When possible, use features that are understandable to humans to aid interpretability

Conclusion

Logistic regression is a powerful yet simple algorithm for binary and multiclass classification. It is widely used in industry and academia for its strong performance, interpretability, and efficiency. However, it is not a silver bullet and has limitations such as linearity assumption and sensitivity to outliers.

In this article, we covered the key concepts of logistic regression including the mathematical formulation, assumptions, evaluation metrics, regularization, and multiclass extensions. We also discussed best practices and industry case studies.

Logistic regression should be one of the first models you reach for when faced with a classification problem. It provides a strong baseline and interpretable results. By following best practices and carefully tuning the model, logistic regression can achieve excellent performance on many real-world problems.