Understanding Linear Regression: A Deep Dive into the Mathematics for AI and Machine Learning

Linear regression is a core concept in machine learning and artificial intelligence. It is one of the most widely used algorithms for supervised learning problems where the goal is to predict a continuous value output. Linear regression has a long history dating back to the early 19th century and the pioneering work of Legendre and Gauss on the method of least squares. Today, linear regression is used across many domains such as:

• Estimating sales and demand forecasts
• Predicting housing and stock prices
• Analyzing sensor data and fault detection
• Estimating health risks and drug responses
• Understanding marketing promotion effects

While modern machine learning has progressed to more advanced algorithms like deep neural networks, linear regression remains an important foundational technique. It is often the first model to try for a supervised learning problem due to its simplicity, interpretability, and computational efficiency. Linear regression also has a well-established statistical theory behind it.

In this article, we will take a deep dive into the mathematics behind linear regression from a machine learning perspective. We‘ll cover the key concepts, equations, and algorithms commonly used in applying linear regression for predictive modeling. Whether you‘re a data scientist, AI researcher, or ML engineer, having a solid grasp of these mathematical foundations will help you effectively leverage linear regression in your work.

Simple Linear Regression

The basic form of linear regression is simple linear regression, where we have a single independent variable (also known as a feature or predictor) and a single dependent variable (the target or outcome). Mathematically, we can express a linear relationship between the variables as:

y = β₀ + β₁x + ε

Where:

• y is the dependent variable
• x is the independent variable
• β₀ is the y-intercept or bias term
• β₁ is the slope or weight that represents the change in y for a 1 unit change in x
• ε is the error term representing the difference between the predicted and actual y values (also known as noise or irreducible error)

The goal of linear regression is to find the values of the model parameters (β₀ and β₁) that minimize the differences between the predicted and actual y values across all data points. In other words, we want to find the line of best fit for the data.

Suppose we have the following training data with a single feature x and target y:

x	y
2.5	18.9
4.2	27.5
1.9	15.3
3.7	25.1
5.1	35.8

We can plot this data and visually fit a line to it:

Mathematically, we can define an objective function J that we want to minimize, which represents the average squared difference between the predicted and actual y values:

J(β₀,β₁) = (1/2m) Σ (ŷ⁽ⁱ⁾ – y⁽ⁱ⁾)²
= (1/2m) Σ (β₀ + β₁x⁽ⁱ⁾ – y⁽ⁱ⁾)²

Where:

• m is the number of training examples
• ŷ⁽ⁱ⁾ is the predicted value for the ith example using the model‘s parameters
• y⁽ⁱ⁾ is the actual value for the ith example
• x⁽ⁱ⁾ is the feature value for the ith example

This objective function is known as the mean squared error (MSE) cost function. Our goal is to find the values of β₀ and β₁ that minimize MSE across all m training examples.

Ordinary Least Squares

One common approach to finding the optimal model parameters is Ordinary Least Squares (OLS). In OLS, we take the partial derivatives of the cost function J with respect to β₀ and β₁, set them equal to zero, and solve the resulting system of linear equations.

∂J/∂β₀ = (1/m) Σ (β₀ + β₁x⁽ⁱ⁾ – y⁽ⁱ⁾) = 0
∂J/∂β₁ = (1/m) Σ (β₀ + β₁x⁽ⁱ⁾ – y⁽ⁱ⁾)x⁽ⁱ⁾ = 0

Solving these equations leads to the following closed-form solution for the optimal parameters:

β₁ = (Σ(x⁽ⁱ⁾ – mean(x))(y⁽ⁱ⁾ – mean(y))) / Σ(x⁽ⁱ⁾ – mean(x))²
β₀ = mean(y) – β₁ * mean(x)

Where mean(x) and mean(y) are the sample means of the x and y values.

Applying this to our example data, we get:

• mean(x) = 3.48
• mean(y) = 24.52
• β₁ = 6.39
• β₀ = 2.27

So our fitted regression equation is:

ŷ = 2.27 + 6.39x

We can assess the fit of the model by looking at metrics like the coefficient of determination (R²), which represents the proportion of variance in y that is predictable from x:

R² = 1 – (SSR / SST)

Where:

• SSR is the sum of squared residuals Σ(y⁽ⁱ⁾ – ŷ⁽ⁱ⁾)²
• SST is the total sum of squares Σ(y⁽ⁱ⁾ – mean(y))²

R² ranges from 0 to 1, with values closer to 1 indicating a better fit. For our example, R² is 0.968, suggesting the linear model captures most of the variance in y.

Gradient Descent

An alternative to the closed-form OLS solution is gradient descent, an iterative optimization algorithm. Gradient descent starts with initial guesses for the parameters and repeatedly takes steps in the direction that minimizes the cost function.

The update equations for the parameters at each iteration are:

β₀ := β₀ – α ∂J/∂β₀
β₁ := β₁ – α ∂J/∂β₁

Where α is the learning rate that controls the size of the steps.

Here‘s an example of applying gradient descent to our data:

def cost_function(X, y, B0, B1):
    m = len(y)
    J = (1/(2*m)) * np.sum((B0 + B1*X - y)**2)
    return J

def gradient_descent(X, y, B0, B1, alpha, iterations):
    m = len(y)
    cost_history = [0] * iterations

    for iteration in range(iterations):
        h = B0 + B1 * X
        cost = cost_function(X, y, B0, B1)
        cost_history[iteration] = cost

        B0_d = (1/m) * np.sum(h - y)
        B1_d = (1/m) * np.dot((h - y), X)  
        B0 -= alpha * B0_d
        B1 -= alpha * B1_d

    return B0, B1, cost_history

Running gradient descent for 10000 iterations with a learning rate of 0.01 produces the following trace of the cost and parameters:

Iteration 0: Cost 449.13, B0 5.71, B1 4.56 
Iteration 1000: Cost 26.68, B0 2.43, B1 6.30
Iteration 2000: Cost 26.46, B0 2.30, B1 6.37
...
Iteration 9000: Cost 26.45, B0 2.27, B1 6.39
Iteration 10000: Cost 26.45, B0 2.27, B1 6.39

We can see gradient descent converges to the same optimal parameters as the OLS solution, just taking many small steps to get there.

Multiple Linear Regression

Simple linear regression can be extended to include multiple independent variables or features:

y = β₀ + β₁x₁ + β₂x₂ + … + βₚxₚ + ε

Where:

• y is the dependent variable
• x₁, x₂, …, xₚ are the p independent variables or features
• β₀, β₁, β₂, …, βₚ are the parameters or coefficients
• ε is the error term

In matrix notation, we have:

y = Xβ + ε

Where:

• y is an n x 1 vector of the target values
• X is an n x (p+1) matrix of the p features values for each of n examples, with an additional column of 1‘s for the intercept
• β is a (p+1) x 1 vector of the coefficients
• ε is an n x 1 vector of the errors

The OLS solution for the coefficients in multiple regression is:

β = (XᵀX)⁻¹Xᵀy

Where Xᵀ is the transpose of X and ⁻¹ represents matrix inversion.

Some key considerations in applying multiple linear regression include:

• Multicollinearity: Having highly correlated features in the model can lead to unstable parameter estimates. One solution is to use regularization techniques like ridge regression or lasso that constrain the coefficient values. Detecting multicollinearity can be done by looking at the variance inflation factor (VIF).

• Model selection: With many potential features to include, it‘s important to choose a subset that balances performance and parsimony. Strategies include forward selection, backward elimination, and stepwise regression. Information criteria like AIC and BIC can help evaluate model trade-offs.

• Interactions and nonlinearity: The basic multiple regression model assumes additive, linear relationships between the features and target. But often there are interactions between features (where the effect of one feature depends on the value of another) and nonlinear effects. These can be accommodated by adding interaction terms (x₁ * x₂) and polynomial terms (x₁², x₁³) as additional features.

• Dummy variables: Multiple regression requires all input features to be numeric. Categorical variables need to be converted to dummy or indicator variables first.

Model Evaluation

Evaluating the performance of linear regression models is important for understanding their limitations and making good predictions on new data. Some key diagnostics include:

• Residual plots: Plotting the residuals (y – ŷ) vs. the predicted values (ŷ) can reveal patterns like non-linearity, heteroscedasticity (non-constant variance), and outliers. Ideally, residuals should look randomly scattered around 0.

• Normal probability plot of residuals: This checks if the residuals are normally distributed, an assumption of many statistical tests. Departures from a straight diagonal line indicate non-normality.

• Partial regression plots: Plotting the residuals of the target vs. each feature can show if the relationships are linear. These plots adjust or control for the effects of the other variables in the model.

• Train/test split: Evaluating the model on held-out test data that wasn‘t used during training gives a more realistic estimate of out-of-sample performance and guards against overfitting. Cross-validation takes this further by averaging the scores across multiple random splits.

• Regularization: Techniques like ridge regression and lasso can help prevent overfitting by shrinking the coefficient values towards zero, especially with a large number of features. The regularization strength is tuned by a hyperparameter (λ).

Conclusion

Linear regression is a fundamental concept in machine learning and AI with a rich mathematical foundation. In this article, we took a deep dive into the key equations and techniques behind applying linear regression for predictive modeling.

We covered the basic form of simple linear regression and deriving the ordinary least squares solution for the optimal model parameters. We then looked at gradient descent as an alternative optimization algorithm.

Extending to multiple linear regression, we discussed key considerations like multicollinearity, model selection, interactions, and dummy encoding. Finally, we reviewed important regression diagnostics and evaluation methods.

Understanding these mathematical concepts is crucial for data scientists and AI/ML practitioners who want to effectively apply linear regression and interpret its results. Linear regression is also an important building block for more advanced methods like generalized linear models, regression trees, and neural networks.

To learn more, check out the following resources:

• An Introduction to Statistical Learning (book)
• Linear Regression (scikit-learn documentation)
• Coursera Machine Learning Course (Andrew Ng)