Demystifying Linear Regression: A Mathematical Perspective with Python

Linear regression is one of the most fundamental and widely used machine learning algorithms. It models the relationship between input and output variables as a linear equation, allowing us to make predictions. While linear regression is conceptually simple, understanding the underlying math is key to effectively leveraging it to solve real-world problems.

In this article, we‘ll take an in-depth look at the mathematical foundation of linear regression and implement it step-by-step in Python. Whether you‘re a data science student, ML practitioner, or just curious about how linear regression works under the hood, this guide will walk you through it all!

The Linear Equation

At the core of linear regression is the simple linear equation you likely learned in algebra class:

y = mx + b

Where:

• y is the output/dependent variable we‘re trying to predict
• x is the input/independent variable
• m is the slope of the line (also called the coefficient)
• b is the y-intercept

In ML terms, we call the inputs "features" and the outputs "labels" or "targets". The goal is to find the optimal values of m and b that define a line that best fits the training data. This line is called the "line of best fit" or "regression line".

Here‘s an example of a linear regression line fit to a dataset:

[Image of linear regression line]

Finding the Line of Best Fit

To find the optimal values for the slope (m) and y-intercept (b), we define an error function (also called a cost or loss function) that measures how far off our model‘s predictions are from the actual values.

A common error function is the mean squared error (MSE), which averages the squared differences between the predicted and actual values:

MSE = (1/n) * Σ(actual – predicted)^2

We use the square of the errors to treat positive and negative differences the same and to penalize larger errors more heavily. The goal is to find the m and b values that minimize the MSE.

To do this, we use a technique called gradient descent. It starts with random values for m and b, then iteratively adjusts them in the direction that reduces the error. It does this by finding the gradient (partial derivatives) of the error function with respect to m and b.

The gradients (partial derivatives) of MSE are:

∂MSE/∂m = (-2/n) Σ(actual – predicted) input

∂MSE/∂b = (-2/n) * Σ(actual – predicted)

The gradient descent algorithm updates m and b each iteration like:

m = m – α * ∂MSE/∂m

b = b – α * ∂MSE/∂b

Where α is the learning rate that controls the size of the updates. This continues until the error stops decreasing significantly (converges).

Here‘s a visualization of the gradient descent process:

[Image/animation of gradient descent]

Evaluating the Model

After fitting the linear regression model, we need to evaluate how well it performs. Some common evaluation metrics include:

• Mean absolute error (MAE): The average of the absolute differences between predicted and actual values. Gives an idea of the average magnitude of errors.

• Root mean squared error (RMSE): The square root of the MSE defined earlier. Penalizes larger errors more heavily than MAE.

• R-squared (coefficient of determination): Measures the proportion of variance in the output that is predictable from the input. Values range from 0 to 1, with 1 being a perfect fit.

R^2 = 1 – (SS_res / SS_tot)

Where SS_res is the sum of squared residuals (actual – predicted) and SS_tot is the total sum of squared differences from the mean output value.

Implementing in Python

Now let‘s see how to implement linear regression in Python using the popular scikit-learn library. We‘ll use a simple example of predicting housing prices based on square footage.

First we‘ll import the required libraries and generate some sample data:

from sklearn.linear_model import LinearRegression
import numpy as np

# Generate random data
num_samples = 100
rng = np.random.RandomState(42)
x = rng.random_sample((num_samples, 1))
y = 50 * x + 10 + rng.normal(size=num_samples) 

Next we‘ll create the linear regression model object and fit it to the data:

model = LinearRegression()
model.fit(x, y)

We can inspect the learned model coefficients:

print(model.coef_) # Slope 
print(model.intercept_) # Intercept

To make predictions on new data:

x_new = np.array([[0.5], [0.8]])
y_pred = model.predict(x_new)

And to evaluate the model‘s performance on the training data:

from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error

y_pred = model.predict(x)
print("R-squared:", r2_score(y, y_pred))
print("MAE:", mean_absolute_error(y, y_pred))  
print("RMSE:", np.sqrt(mean_squared_error(y, y_pred)))

[Image of pairwise scatterplots]

This covers a basic example of simple linear regression with one input variable. The process is very similar for multiple linear regression with multiple input features. The model equation becomes:

y = m1x1 + m2x2 + … + b

And the coefficient updates in gradient descent are done separately for each mi.

Assumptions and Limitations

It‘s important to be aware of the assumptions linear regression makes about the data and the limitations they impose. The main ones are:

1. Linearity: The relationship between inputs and output must be linear.

2. Independence: The input features must be independent of each other (no multicollinearity).

3. Homoscedasticity: The variance of the errors must be constant across all input values.

4. Normality: The errors must follow a normal (Gaussian) distribution.

If these assumptions are violated, the results of linear regression can be unreliable or misleading. Ways to address these issues include:

• Transforming nonlinear data (e.g. logarithms)
• Removing correlated features
• Using regularization techniques like ridge regression or lasso
• Trying alternative models like polynomial regression

Wrapping Up

We covered a lot of ground in this guide to the math behind linear regression! To recap:

• Linear regression models the input-output relationship linearly with coefficients for the slope and intercept
• The line of best fit is found by minimizing the mean squared error cost function
• Gradient descent iteratively updates the coefficients in the direction that reduces the error
• Scikit-learn makes it straightforward to implement linear regression models in a few lines of Python code
• Linear regression has assumptions that must be met to produce reliable results

I hope this gives you a solid foundation for understanding how linear regression works and the ability to implement it in your own projects. Let me know if you have any other questions!