A Beginner‘s Guide to Logistic Regression Using Python

1. Introduction

Welcome aspiring data scientist! In this tutorial, we‘ll dive into one of the fundamental machine learning algorithms – logistic regression. While linear regression predicts continuous numerical values, logistic regression is used for classification tasks with categorical target variables. We‘ll uncover the inner workings of logistic regression, when to use it, and how to implement it in Python step-by-step. Get ready to level up your machine learning skills!

2. What is Logistic Regression?

Logistic regression is a popular algorithm for solving binary classification problems, where the target variable has only two possible outcomes, typically represented as 0 or 1. It estimates the probability of an instance belonging to a particular class. If the estimated probability is greater than a specified threshold (usually 0.5), the model assigns the instance to class 1, otherwise to class 0.

Some common applications of logistic regression include:

• Email spam detection (spam or not spam)
• Churn prediction (customer stays or leaves)
• Disease diagnosis (has disease or not)
• Loan default prediction (defaults or pays back)

Despite its name, logistic regression is actually a classification algorithm, not regression. It got its name because it uses a logistic function under the hood.

3. Mathematics of Logistic Regression

At the core of logistic regression lies the sigmoid function, also known as the logistic function:

$\sigma(z) = \frac{1}{1+e^{-z}}$

where $z$ is the linear combination of input features and their coefficients:

$z = \beta_0 + \beta_1x_1 + \beta_2x_2 + … + \beta_nx_n$

The sigmoid function squashes the value of $z$ to a probability between 0 and 1. This probability represents the likelihood of the instance belonging to class 1.

To train a logistic regression model, we need to find the optimal values of the coefficients $\beta_0, \beta_1, …, \beta_n$ that minimize the cost function. The cost function measures the difference between the predicted probabilities and the actual class labels. A common choice is the log loss function:

$J(\beta) = -\frac{1}{m}\sum_{i=1}^m [y^{(i)} \log(h(x^{(i)})) + (1-y^{(i)}) \log(1-h(x^{(i)}))]$

where $m$ is the number of training instances, $y^{(i)}$ is the actual class label of instance $i$, and $h(x^{(i)})$ is the predicted probability of instance $i$ belonging to class 1.

To minimize the cost function and find the optimal coefficients, we use an optimization algorithm like gradient descent. It iteratively updates the coefficients in the direction of steepest descent of the cost function until convergence.

4. Types of Logistic Regression Problems

Logistic regression can be applied to different types of classification problems:

1. Binary Logistic Regression: The target variable has only two possible outcomes (e.g., yes/no, true/false, 0/1).

2. Multinomial Logistic Regression: The target variable has three or more unordered categories (e.g., food type: pizza, burger, taco).

3. Ordinal Logistic Regression: The target variable has three or more ordered categories (e.g., movie ratings: 1 star, 2 stars, 3 stars, 4 stars, 5 stars).

In this tutorial, we‘ll focus on binary logistic regression.

5. Logistic Regression with Categorical Variables

Logistic regression assumes a linear relationship between the input features and the log-odds of the target variable. However, categorical features cannot be directly used in the model since they are non-numeric. We need to encode them into numerical form first.

For nominal categorical variables (unordered categories), we use one-hot encoding. It creates a new binary feature for each category and assigns a value of 1 to the feature corresponding to the instance‘s category, and 0 to the rest. For example, if we have a "color" feature with categories "red", "green", and "blue", one-hot encoding will create three new features: "color_red", "color_green", and "color_blue".

For ordinal categorical variables (ordered categories), we can use ordinal encoding. It assigns a unique integer value to each category based on its order. For instance, for the "size" feature with categories "small", "medium", and "large", we can assign the values 1, 2, and 3 respectively.

6. Interpreting Logistic Regression Output

One advantage of logistic regression is its interpretability. The coefficients of the model can provide insights into the impact of each feature on the predicted probability.

• A positive coefficient indicates that an increase in the corresponding feature value increases the likelihood of the instance belonging to class 1.
• A negative coefficient indicates that an increase in the corresponding feature value decreases the likelihood of the instance belonging to class 1.
• The magnitude of the coefficient represents the strength of the feature‘s impact on the prediction.

However, interpreting the coefficients directly can be tricky due to the non-linearity of the sigmoid function. Instead, we can interpret the odds ratios, which are obtained by exponentiating the coefficients. An odds ratio greater than 1 means the feature increases the odds of the instance belonging to class 1, while an odds ratio less than 1 means the feature decreases the odds.

7. Evaluating Logistic Regression Models

To assess the performance of a logistic regression model, we use evaluation metrics derived from the confusion matrix. The confusion matrix summarizes the model‘s predictions versus the actual class labels. It has four components:

• True Positives (TP): Instances correctly predicted as class 1.
• True Negatives (TN): Instances correctly predicted as class 0.
• False Positives (FP): Instances incorrectly predicted as class 1.
• False Negatives (FN): Instances incorrectly predicted as class 0.

From the confusion matrix, we can calculate several evaluation metrics:

• Accuracy: The proportion of correct predictions $(TP+TN)/(TP+TN+FP+FN)$.
• Precision: The proportion of true positive predictions among all positive predictions $(TP)/(TP+FP)$.
• Recall (Sensitivity): The proportion of true positive predictions among all actual positive instances $(TP)/(TP+FN)$.
• F1 Score: The harmonic mean of precision and recall $2 (precisionrecall)/(precision+recall)$.

Another evaluation tool is the Receiver Operating Characteristic (ROC) curve, which plots the true positive rate (recall) against the false positive rate $(FP)/(FP+TN)$ at different classification thresholds. The Area Under the ROC Curve (AUC-ROC) is a metric that summarizes the model‘s performance across all thresholds. An AUC-ROC of 1 indicates a perfect classifier, while 0.5 indicates a random classifier.

8. Regularization in Logistic Regression

Logistic regression is prone to overfitting, especially when the number of features is large compared to the number of training instances. Overfitting occurs when the model learns the noise in the training data and fails to generalize well to unseen data.

To mitigate overfitting, we can apply regularization techniques. Regularization adds a penalty term to the cost function that discourages large coefficient values. The two common types of regularization are:

1. L1 regularization (Lasso): Adds the absolute values of the coefficients to the cost function. It tends to shrink some coefficients to exactly zero, performing feature selection.

2. L2 regularization (Ridge): Adds the squared values of the coefficients to the cost function. It tends to shrink all coefficients towards zero.

The strength of regularization is controlled by a hyperparameter $\lambda$. A higher value of $\lambda$ implies stronger regularization.

9. Python Tutorial: Logistic Regression on Titanic Dataset

Now let‘s put our knowledge into practice by building a logistic regression model to predict passenger survival on the Titanic dataset using Python.

9.1. Libraries and Dataset

First, we‘ll import the necessary libraries and load the Titanic dataset:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, accuracy_score, precision_score, recall_score, f1_score, roc_curve, roc_auc_score

titanic_data = pd.read_csv(‘titanic.csv‘)

9.2. Exploratory Data Analysis

Let‘s explore the dataset to gain insights and identify any data quality issues:

titanic_data.head()
titanic_data.info()
titanic_data.describe()
sns.countplot(x=‘Survived‘, data=titanic_data)
sns.countplot(x=‘Survived‘, hue=‘Sex‘, data=titanic_data)
sns.countplot(x=‘Survived‘, hue=‘Pclass‘, data=titanic_data)

9.3. Data Preprocessing

We‘ll perform data cleaning, handle missing values, and encode categorical features:

titanic_data[‘Age‘].fillna(titanic_data[‘Age‘].median(), inplace=True)
titanic_data[‘Embarked‘].fillna(titanic_data[‘Embarked‘].mode()[0], inplace=True)
titanic_data.drop([‘PassengerId‘, ‘Name‘, ‘Ticket‘, ‘Cabin‘], axis=1, inplace=True)

sex_encoder = {‘male‘: 0, ‘female‘: 1}
titanic_data[‘Sex‘] = titanic_data[‘Sex‘].map(sex_encoder)

embarked_encoder = {‘S‘: 0, ‘C‘: 1, ‘Q‘: 2}
titanic_data[‘Embarked‘] = titanic_data[‘Embarked‘].map(embarked_encoder)

9.4. Training a Model

We‘ll split the data into train and test sets, create a logistic regression model, and train it on the training data:

X = titanic_data.drop(‘Survived‘, axis=1)
y = titanic_data[‘Survived‘]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

logreg = LogisticRegression()
logreg.fit(X_train, y_train)

9.5. Model Evaluation

We‘ll make predictions on the test set and evaluate the model‘s performance using various metrics:

y_pred = logreg.predict(X_test)

print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))

print("Accuracy:", accuracy_score(y_test, y_pred))
print("Precision:", precision_score(y_test, y_pred))
print("Recall:", recall_score(y_test, y_pred))
print("F1 Score:", f1_score(y_test, y_pred))

y_pred_proba = logreg.predict_proba(X_test)[:, 1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred_proba)
plt.plot(fpr, tpr)
plt.xlabel(‘False Positive Rate‘)
plt.ylabel(‘True Positive Rate‘)
plt.title(‘ROC Curve‘)
print("AUC-ROC:", roc_auc_score(y_test, y_pred_proba))

9.6. Improving Model Performance

We can try different techniques to improve the model‘s performance, such as:

• Feature selection: Remove irrelevant or redundant features.
• Regularization: Apply L1 or L2 regularization to prevent overfitting.
• Hyperparameter tuning: Use techniques like grid search or random search to find the optimal hyperparameters.

from sklearn.feature_selection import RFE
from sklearn.model_selection import GridSearchCV

selector = RFE(logreg, n_features_to_select=5, step=1)
selector.fit(X_train, y_train)
X_train_selected = selector.transform(X_train)
X_test_selected = selector.transform(X_test)

param_grid = {‘C‘: [0.001, 0.01, 0.1, 1, 10, 100]}
logreg_cv = GridSearchCV(logreg, param_grid, cv=5)
logreg_cv.fit(X_train_selected, y_train)

print("Best hyperparameters:", logreg_cv.best_params_)

10. Conclusion

Congratulations on making it to the end of this tutorial! You‘ve learned the fundamentals of logistic regression, when to use it, how it works mathematically, and how to implement it in Python. You‘ve also learned techniques to handle categorical variables, evaluate model performance, and improve model performance.

Logistic regression is a powerful and interpretable algorithm for binary classification tasks. However, it has its limitations, such as assuming a linear relationship between the features and the log-odds of the target variable. In cases where this assumption doesn‘t hold, you may need to consider more advanced algorithms like decision trees, random forests, or neural networks.

Remember, the key to mastering machine learning is practice. Try applying logistic regression to different datasets, experiment with various techniques, and compare its performance with other algorithms. Happy learning!