# 5 Regression Algorithms You Should Know – Introductory Guide

- Canonical: https://33rdsquare.com/5-regression-algorithms-you-should-know-introductory-guide/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

Regression is a core technique in the machine learning practitioner‘s toolbox. It allows us to predict continuous numerical outcomes from a set of input features – everything from forecasting sales figures to diagnosing the progression of a disease. A study by Deloitte estimated that by 2025, machine learning and AI could be worth up to $15.7 trillion annually1, with regression being one of the most widely deployed techniques across industries.

In this guide, we‘ll introduce 5 essential regression algorithms every data scientist should know:

1. Linear Regression
2. Polynomial Regression
3. Ridge Regression
4. Lasso Regression
5. Decision Tree Regression

We‘ll dive into the mathematical formulation of each model, interpret their learned parameters, demonstrate how to implement them in Python, and discuss best practices for selecting and evaluating models. By the end, you‘ll have the foundations to apply these techniques to real-world problems and the intuition to know which one to reach for in different scenarios.

## 1. Linear Regression

Linear regression is the classic workhorse of statistics and machine learning. It models the relationship between input features X and a target variable y as a linear function:

$y = \beta_0 + \beta_1x_1 + … + \beta_nx_n$

where $\beta_0$ is the intercept and $\beta_1$ to $\beta_n$ are the coefficients for each feature. Linear regression has an elegant closed-form solution for the optimal coefficients using ordinary least squares:

$\hat{\beta} = (X^TX)^{-1}X^Ty$

However, in practice the coefficients are learned using an optimization algorithm like gradient descent to minimize the mean squared error loss between the predicted and actual y values:

$MSE = \frac{1}{n}\sum_{i=1}^n(y_i – \hat{y}_i)^2$

The learned $\beta$ coefficients directly tell us the expected change in y for a one unit increase in the corresponding x feature, holding all others constant. This interpretability is a big advantage of linear models. For example, in a real estate pricing model, a $\beta$ of 50 for square_feet means we expect a $50,000 increase in price for every 1000 extra square feet, all else equal.

However, linear regression makes several strong assumptions that limit its real-world applicability, namely linearity, independence of features, normality and equal variance of errors, and no multicollinearity. Violating these can lead to poor extrapolation and widely biased coefficients.

Nonetheless, linear regression is widely used in practice for problems like:

- Predicting crop yields based on rainfall, temperature, fertilizer, etc.
- Forecasting electricity demand based on economic and demographic factors
- Diagnosing diabetes progression based on patient biomarkers

**Python Implementation:**

```
from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X_train, y_train)

print(f"Learned coefficients: {model.coef_}")

y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
print(f"Test MSE: {mse:.2f}")
```

![Linear Regression Fit](https://33rdsquare.com/linear_regression.png)

## 2. Polynomial Regression

Polynomial regression extends linear regression to capture non-linear relationships between features and the target variable. It does this by creating new features that are powers of the original features, then fitting a linear model on this expanded feature set.

For example, a 2nd-degree polynomial with two features would be:

$y = \beta_0 + \beta_1x_1 + \beta_2x_1^2 + \beta_3x_2 + \beta_4x_2^2 + \beta_5x_1x_2$

Polynomial regression can approximate arbitrary non-linear functions given a high enough degree. However, this flexibility comes at a cost of increased model complexity, slower training times, and a higher risk of overfitting, especially in low data regimes. Regularization becomes critical for polynomial models.

Still, polynomial features are an easy way to improve model performance when we expect non-linear relationships. They work well in combination with other feature engineering techniques like binning or spline transformations. Some common use cases are:

- Modeling the non-linear relationship between drug dosage and patient response
- Predicting customer churn based on metrics like purchase history and engagement
- Analyzing sensor time series data for predictive maintenance of equipment

**Python Implementation:**

```
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

model = make_pipeline(
  PolynomialFeatures(degree=2),
  LinearRegression()
)

model.fit(X_train, y_train)
print(f"Test R^2: {model.score(X_test, y_test):.2f}")
```

![Polynomial Regression Fit](https://33rdsquare.com/polynomial_regression.png)

## 3. Ridge Regression

Ridge regression is a regularized version of linear regression that constrains the magnitude of the learned coefficients. Specifically, it adds an L2 penalty on the coefficients to the ordinary least squares objective:

$\min_\beta \sum_{i=1}^n(y_i – \hat{y}_i)^2 + \alpha\sum_{j=1}^p\beta_j^2$

Here $\alpha$ is a hyperparameter that controls the strength of regularization. As $\alpha$ increases, the coefficients are shrunk towards zero, trading off bias for variance. This reduces overfitting, especially when we have a large number of features.

The ridge coefficients have a convenient closed-form solution:
 $\hat{\beta} = (X^TX + \alpha I)^{-1}X^Ty$

Ridge works well when we expect there to be many features that each weakly predict the target. For example, in genetics problems we often have thousands of gene expressions and SNPs which each marginally influence a phenotype.

Unlike lasso regression (covered next), ridge doesn‘t perform feature selection, and will keep all features in the model, just scaled down. This can make interpreation more difficult vs. a sparse solution.

Some other downsides of ridge are that it requires careful normalization of features and target, and it‘s more sensitive to outliers vs. robust algorithms like Huber regression2.

**Python Implementation:**

```
from sklearn.linear_model import Ridge

model = Ridge(alpha=0.1)
model.fit(X_train, y_train)

print(f"Number of coefficients: {np.sum(model.coef_ != 0)}")
```

![Ridge Path](https://33rdsquare.com/ridge_path.png)

## 4. Lasso Regression

Lasso (Least Absolute Shrinkage and Selection Operator) regression is another regularized linear regression model. However, instead of an L2 penalty on the coefficients, it imposes an L1 penalty:

$\min_\beta \sum_{i=1}^n(y_i – \hat{y}_i)^2 + \alpha\sum_{j=1}^p|\beta_j|$

The L1 penalty causes some coefficients to be set exactly to zero for sufficiently large values of $\alpha$. This means lasso automatically performs feature selection! The sparsity of lasso solutions make them more interpretable and cheaper to collect/store inputs for than dense models.

Lasso is the go-to model when we believe only a few features are relevant and we want to identify them. It‘s often used in high-dimensional settings like text classification with bag-of-words features or genomic analysis3.

Lasso does have some limitations. If features are highly correlated, lasso tends to pick one at random, while ridge shrinks them together. Lasso also doesn‘t handle multicollinearity well and can produce unstable solutions.

A popular extension is elastic net regression, which combines both L1 and L2 penalties, getting the feature selection of lasso and the coefficient stability of ridge.

**Python Implementation:**

```
from sklearn.linear_model import Lasso

model = Lasso(alpha=0.01)
model.fit(X_train, y_train)

print(f"Number of features used: {np.sum(model.coef_ != 0)}")
```

![Lasso Path](https://33rdsquare.com/lasso_path.png)

## 5. Decision Tree Regression

Decision trees take a completely different approach to regression than the linear models we‘ve covered so far. They work by recursively partitioning the input space into disjoint regions, then predicting a constant value within each region.

The tree is grown greedily by choosing the best split point that minimizes the MSE of the resulting child regions. This process is repeated until a stopping criteria is met, such as a maximum depth or minimum number of samples per leaf.

For prediction, the new point is walked down the tree to its corresponding leaf node region, and the mean target value of the training points in that region is returned.

Decision trees are popular because they‘re easy to visualize and interpret, can handle both numerical and categorical features, are insensitive to input scaling, and automatically learn feature interactions. Some common use cases are:

- Predicting customer lifetime value based on demographics and purchase history
- Estimating house prices from property features like size, age, location, etc.
- Forecasting product demand based on historical sales, price, marketing spend, etc.

However, decision trees are notoriously prone to overfitting and have high variance. They often underperform in low-dimensional settings. This has led to ensemble tree methods like random forests and gradient boosting dominating in practice.

**Python Implementation:**

```
from sklearn.tree import DecisionTreeRegressor

model = DecisionTreeRegressor(max_depth=3)
model.fit(X_train, y_train)

print(f"Test MAE: {mean_absolute_error(y_test, model.predict(X_test)):.3f}")
```

![Decision Tree Regression](https://33rdsquare.com/decision_tree_regression.png)

## Choosing Among Regression Algorithms

With so many regression algorithms at our disposal, how do we know which one to use in practice? While there are no hard and fast rules, here are some general guidelines:

- Linear regression is a good baseline and interpretable in low-dimensional settings
- Polynomial regression extends linear models to capture simple non-linear relationships
- Ridge regression is effective when we have many weakly predictive features
- Lasso regression is ideal for automatic feature selection and model interpretability
- Decision trees are flexible and work well with heterogeneous, unscaled features

Ultimately, the best model depends on the specifics of your data and prediction task. It‘s common to evaluate multiple algorithms and let empirical performance guide model selection.

Some additional factors to consider are:

- Computational resources required for training and inference
- Ability to handle missing data or outliers
- Scalability to large datasets and high-dimensional feature spaces
- Hyperparameter tuning and validation strategies
- Consistency and calibration of uncertainty estimates

It‘s also important to remember that choosing the algorithm is just one step in the larger process of constructing an effective ML pipeline. The concept of "no free lunch" tells us that no single model will be best for every problem4.

Significant performance gains often come from better feature engineering, cross-validation and hyperparameter tuning, ensembling, and cleaner, larger datasets. The algorithms covered here provide a foundation, but should be complemented with rigorous statistical practices and domain expertise.

## Trends and Future Directions

While these 5 algorithms represent the core of regression methods, the field is rapidly evolving. Some notable advancements include:

- Gaussian Process Regression, which provides uncertainty estimates and a principled approach to model selection5
- Quantile regression, which predicts a given quantile of the target variable and is robust to outliers6
- Deep learning models like Neural Networks, which can learn complex non-linear relationships and scale to massive datasets
- Automated Machine Learning (AutoML) tools for model selection, hyperparameter optimization, and feature engineering7

As datasets grow in size and models in complexity, we‘re also seeing a shift from traditional statistical models to more scalable machine learning algorithms. This has created a need for better model governance, interpretability, and bias testing.

Exciting applications are emerging at the intersection of economics, statistics, and machine learning, from estimating heterogeneous treatment effects8 to inferring causal relationships from observational data9. As these techniques mature, we can expect regression to continue enabling data-driven insights and powering critical business decisions across industries.

## References

[1] Deloitte. (2019). Leveraging AI to Transform the Enterprise. [https://www2.deloitte.com/content/dam/Deloitte/global/Documents/About-Deloitte/gx-ai-transformation-report.pdf](https://www2.deloitte.com/content/dam/Deloitte/global/Documents/About-Deloitte/gx-ai-transformation-report.pdf)

[2] Huber, P. J. (1992). Robust estimation of a location parameter. In Breakthroughs in statistics (pp. 492-518). Springer, New York, NY.

[3] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267-288.

[4] Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1), 67-82.

[5] Rasmussen, C. E. (2003, February). Gaussian processes in machine learning. In Summer school on machine learning (pp. 63-71). Springer, Berlin, Heidelberg.

[6] Koenker, R., & Hallock, K. F. (2001). Quantile regression. Journal of economic perspectives, 15(4), 143-156.

[7] Yao, Q., Wang, M., Escalante, H. J., Guyon, I., Hu, Y. Q., Li, Y. F., … & Viegas, E. (2018). Taking human out of learning applications: A survey on automated machine learning. arXiv preprint arXiv:1810.13306.

[8] Athey, S., & Imbens, G. W. (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27), 7353-7360.

[9] Pearl, J. (2009). Causal inference in statistics: An overview. Statistics surveys, 3, 96-146.

---

Source: [5 Regression Algorithms You Should Know – Introductory Guide](https://33rdsquare.com/5-regression-algorithms-you-should-know-introductory-guide/)