Introduction to Linear Predictive Models – Part 2

Welcome back to our two-part series on linear predictive models! In Part 1, we covered the fundamentals of simple linear regression for predicting a continuous outcome variable from a single predictor variable. We learned how to fit a regression line, assess model fit, and make predictions on new data.

In Part 2, we‘ll expand our scope to more advanced linear predictive models that can handle multiple predictor variables and combat common issues like overfitting and correlated features. Specifically, we‘ll dive into a family of techniques called regularized regression, which encompasses models like ridge regression, lasso regression, and elastic net.

Along the way, we‘ll also introduce the concept of a linear predictor, discuss connections to generalized linear models (GLMs), and provide a Python code walkthrough on implementing regularized regression. Let‘s get started!

A Primer on Linear Predictors

Before we jump into specific linear predictive models, it‘s important to understand the general concept of a linear predictor. In short, a linear predictor is a weighted sum of the input features used to predict the outcome variable in a linear model.

Mathematically, for a set of p predictor variables X1, X2, …, Xp, the linear predictor η is defined as:

η = β0 + β1X1 + β2X2 + … + βpXp

Here, β0 is the intercept term and β1, β2, …, βp are the coefficients that determine the effect of each predictor variable on the outcome. The coefficients are estimated during model fitting to minimize a loss function, typically the mean squared error (MSE) between the true and predicted outcome values.

The linear predictor is a key component of many statistical models beyond just linear regression. In generalized linear models (GLMs), the linear predictor is linked to the expected value of the outcome variable through a link function g():

g(E[Y]) = η = β0 + β1X1 + β2X2 + … + βpXp

Different choices of link function give rise to various GLMs suitable for outcomes with different distributions, like logistic regression for binary outcomes or Poisson regression for count data. Understanding the linear predictor is crucial for working with these more advanced models.

Regularized Regression Models

Now let‘s turn our attention to regularized regression models, which add constraints on the magnitude of the coefficient estimates to prevent overfitting and handle correlated features. The two most popular types of regularized regression are ridge regression and lasso regression.

Ridge Regression

Ridge regression, also known as L2 regularization, adds a penalty term to the ordinary least squares (OLS) loss function that is proportional to the square of the L2 norm of the coefficient vector:

minimize (MSE + λ * Σ(βj^2))

The hyperparameter λ ≥ 0 controls the strength of the penalty – larger values of λ result in greater shrinkage of the coefficient estimates towards zero. The L2 penalty has the effect of evenly spreading out the impact of correlated features, as it prefers smaller, more diffuse coefficient values.

Notably, ridge regression does not perform feature selection, as it keeps all predictors in the model (unless λ is very large). It is best suited for situations with many correlated features where we don‘t want to eliminate any entirely.

Lasso Regression

Lasso regression, short for least absolute shrinkage and selection operator, modifies the loss function to use an L1 penalty on the coefficient vector:

minimize (MSE + λ * Σ(|βj|))

Unlike ridge, the L1 penalty allows coefficient estimates to be shrunken exactly to zero, effectively performing feature selection. As λ increases, more coefficients will be eliminated from the model entirely. Lasso tends to select one feature among a group of correlated features, making it ideal for seeking sparse, interpretable models.

Elastic Net

Elastic net regression strikes a balance between ridge and lasso by combining both L1 and L2 penalties:

minimize (MSE + λ1 Σ(|βj|) + λ2 Σ(βj^2))

The ratio of λ1 to λ2 controls the balance between lasso-like sparsity and ridge-like stability. Elastic net is a good choice when you have a large number of correlated features and want to maintain some sparsity in the model.

Python Code Example

To illustrate regularized regression in action, let‘s walk through a Python code example using the scikit-learn library. We‘ll fit ridge and lasso regression models to the Boston Housing dataset and compare their performance.

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge, Lasso
from sklearn.metrics import mean_squared_error

# Load Boston Housing dataset
boston = load_boston()
X, y = boston.data, boston.target

# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Standardize features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Fit ridge regression model
ridge = Ridge(alpha=1.0)
ridge.fit(X_train_scaled, y_train)

# Make predictions on test set and compute MSE
y_pred_ridge = ridge.predict(X_test_scaled)
mse_ridge = mean_squared_error(y_test, y_pred_ridge)
print(f"Ridge MSE: {mse_ridge:.2f}")

# Fit lasso regression model 
lasso = Lasso(alpha=0.1)
lasso.fit(X_train_scaled, y_train) 

# Make predictions and compute MSE
y_pred_lasso = lasso.predict(X_test_scaled)
mse_lasso = mean_squared_error(y_test, y_pred_lasso)  
print(f"Lasso MSE: {mse_lasso:.2f}")

In this example, we first load the Boston Housing dataset and split it into train and test sets. We standardize the features using StandardScaler to ensure they are on the same scale.

Next, we fit a ridge regression model with alpha=1.0 (equivalent to λ) and a lasso model with alpha=0.1. We make predictions on the test set and print the mean squared error for each model.

The key hyperparameter in both ridge and lasso is the regularization strength alpha. Optimal values can be found using techniques like cross-validation and grid search over a range of alpha values. Scikit-learn provides utilities like RidgeCV and LassoCV to automate this tuning process.

Extensions and Variants

Beyond basic regularized linear regression, there are many extensions and variants worth mentioning:

• Polynomial regression extends linear models by adding interaction terms and higher-order powers of the input features to capture non-linear relationships. However, this can quickly lead to high-dimensional, overfitted models.

• Generalized additive models (GAMs) provide a more flexible approach by modeling each feature using smooth, nonlinear functions and summing their contributions. Modern GAM software like pyGAM and mgcv allow fitting complex, interpretable models while controlling overfitting.

• Quantile regression estimates conditional quantiles of the outcome variable rather than just the mean, providing a more complete picture of the relationship between the predictors and outcome. This is useful for understanding the spread and shape of the conditional outcome distribution.

Concluding Thoughts

We‘ve now seen how regularized regression models like ridge, lasso, and elastic net extend simple linear regression to handle multiple correlated features and prevent overfitting. By adding constraints on the coefficient estimates, these methods provide a powerful toolkit for predictive modeling.

When faced with a linear predictive modeling task, consider the following guidelines:

• If you have a small number of uncorrelated features and don‘t expect overfitting, start with simple linear regression.
• If you have many correlated features and want to keep them all in the model, use ridge regression.
• If you want to perform feature selection and obtain a sparse, interpretable model, use lasso regression.
• If you have a large number of correlated features and want a balance between sparsity and stability, use elastic net.
• If you suspect non-linear relationships, consider polynomial regression or generalized additive models.

Of course, these are just general recommendations – always experiment with multiple approaches and thoroughly validate performance on held-out test data. I hope this two-part series has given you a solid foundation for understanding and applying linear predictive models. Happy modeling!