Load data
Linear regression is one of the most fundamental and widely used machine learning algorithms, employed for a variety of predictive modeling tasks. In its basic form, linear regression fits a line to a set of training data by minimizing the mean squared error between the predicted and actual target values.
However, a common problem with this classic approach is that the model can easily overfit the training data, especially when dealing with a large number of input features. Overfitting means the model captures noise in the training set and fails to generalize well to new, unseen data. The result is poor predictive performance.
This is where regularization comes to the rescue. Regularization techniques constrain the complexity of the model by adding a penalty term to the loss function that encourages smaller coefficient values. This helps combat overfitting and often leads to better generalization and lower test error. Let‘s take a closer look at how regularization works and compare some popular regularized regression methods.
Understanding Regularization
In regularized linear regression, the standard mean squared error loss function is augmented with an additional regularization term:
Here, is the regularization function that penalizes large coefficient values,
is a hyperparameter that controls the strength of regularization,
is the true target value, and
is the model‘s prediction.
The two most common choices for are:
- L1 regularization (Lasso):
- L2 regularization (Ridge):
L1 regularization adds the sum of the absolute values of the coefficients to the loss, while L2 uses the sum of squared values. Both have the effect of shrinking the coefficients, but in slightly different ways as illustrated below:
Lasso tends to drive some coefficients to exactly zero, performing automatic feature selection, while ridge regression just makes coefficients smaller without explicit sparsity.
A third option is Elastic Net regression, which linearly combines both L1 and L2 penalties, providing a compromise between the two.
Evaluating Regularized vs Unregularized Regression
To compare the performance of these different approaches, let‘s work through an example using the Boston Housing dataset which contains information about homes in the Boston area and the goal is to predict the median house price.
First, we load the data, split into train and test sets, and fit the four models:
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Lasso, Ridge, ElasticNet
from sklearn.metrics import mean_squared_error
boston = load_boston()
X, y = boston.data, boston.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
lr = LinearRegression().fit(X_train, y_train)
lasso = Lasso().fit(X_train, y_train)
ridge = Ridge().fit(X_train, y_train)
elastic = ElasticNet().fit(X_train, y_train)
Next, we calculate the train and test mean squared error for each:
models = [lr, lasso, ridge, elastic]
model_names = [‘Linear Regression‘, ‘Lasso‘, ‘Ridge‘, ‘ElasticNet‘]
for model, name in zip(models, model_names):
train_error = mean_squared_error(y_train, model.predict(X_train))
test_error = mean_squared_error(y_test, model.predict(X_test))
print(f"{name}:")
print(f" Train Error: {train_error:.2f}")
print(f" Test Error: {test_error:.2f}")
Linear Regression: Train Error: 21.51 Test Error: 30.80 Lasso: Train Error: 29.69 Test Error: 29.61 Ridge: Train Error: 29.45 Test Error: 28.89 ElasticNet: Train Error: 29.91 Test Error: 30.38
The results show that the unregularized linear regression model achieves the lowest training error but has significantly higher test error, a clear indication of overfitting. The regularized models have slightly higher train error but this is more than compensated for by the much better test set performance. Ridge regression attains the best overall test error in this case.
We can visualize the differences in coefficient values across the models:
import matplotlib.pyplot as plt
fig, axs = plt.subplots(1, 4, figsize=(12,3))
for i, (model, name) in enumerate(zip(models, modelnames)):
axs[i].stem(model.coef)
axs[i].set_title(name)
axs[i].set_xlim(-0.5, 13.5)
axs[i].set_ylim(-5, 50)
fig.suptitle("Model Coefficients")
fig.tight_layout()
As expected, the regularized models have shrunken coefficient estimates compared to plain linear regression. Lasso performs the harshest shrinkage, with several coefficients driven completely to 0. Ridge and ElasticNet also substantially reduce coefficient magnitudes but retain all variables.
The Importance of Tuning
The hyperparameter determines the balance between fitting the training data and keeping model complexity under control. It requires careful tuning to get the best test set performance. Too high a value will lead to oversimplified models with high bias, while too low a value can still result in some overfitting.
This tuning is typically done through cross-validation on the training set, evaluating a range of values and selecting the one that minimizes the cross-validated error:
from sklearn.linear_model import RidgeCV
from sklearn.model_selection import KFold
kfold = KFold(n_splits=5)
lambdas = 10**np.linspace(-5, 5, 100)
ridge_cv = RidgeCV(alphas=lambdas, cv=kfold).fit(X_train, y_train)
print(f"Best lambda: {ridgecv.alpha:.3f}")
print(f"Test Error: {mean_squared_error(y_test, ridge_cv.predict(X_test)):.3f}")
Best lambda: 9.307 Test Error: 25.756
Performing this hyperparameter search leads to a further improvement in the test error compared to the default used previously. Similar tuning can be done for lasso and elastic net.
Conclusions
Through this analysis, we‘ve seen the substantial benefits of using regularized linear regression models like lasso, ridge, and elastic net compared to standard linear regression. Key takeaways include:
-
Regularization helps to prevent overfitting by penalizing large coefficient values, resulting in simpler models that generalize better to unseen data.
-
Ridge regression tends to outperform lasso in most cases, while elastic net provides a good balance between the two. The best choice depends on the specific dataset and problem.
-
Tuning the regularization hyperparameter
is crucial to getting the optimal test set performance. This should be done via cross-validation.
-
Regularization is a simple yet powerful technique that can be applied to many other machine learning models beyond linear regression.
While unregularized linear regression certainly has its place, incorporating regularization should be the default approach, especially for high-dimensional datasets. The small increase in computational cost is more than worth the significant improvements in model performance and reliability.
I encourage you to try out regularized regression on your own datasets and see the benefits firsthand. Feel free to use the code examples provided as a starting point. Happy modeling!
