Data Science 101: Introduction to Cost Functions in Linear Regression

What is Linear Regression?

Linear regression is a fundamental supervised machine learning algorithm used to model and predict continuous numerical values. The goal of linear regression is to find a linear relationship between one or more input variables (also known as features or predictors) and a single output variable (the target or response).

For example, we could use linear regression to model the relationship between years of work experience and salary. Years of experience would be the input variable, and salary would be the output we are trying to predict. Linear regression would find the straight line that best fits the observed data points.

In mathematical terms, a linear regression model takes the form:

y = β0 + β1×1 + β2×2 + … + βnxn

where:

• y is the predicted output value
• x1, x2, …, xn are the input feature values
• β0 is the y-intercept or bias term
• β1, β2, …, βn are the feature weights or coefficients that determine the slope

The key question is: how do we find the optimal values for the bias and weight terms? This is where cost functions come into play as a way to quantify how well a given line fits the training data.

Introducing Cost Functions

A cost function, also known as a loss function or objective function, is a measure of how wrong the model is in terms of its ability to estimate the relationship between input and output variables. We can think of it as a penalty for a bad prediction – the higher the cost, the worse the model.

Specifically, a cost function calculates the error between the predicted output values from the model and the actual ground truth values from the training dataset. The goal of training a linear regression model (or really any machine learning model) is to find the model parameters that minimize this cost function.

Some desirable properties of a cost function are:

• Convex: A convex function has only one global minimum, making optimization easier. Most commonly used cost functions are convex.
• Differentiable: We need to be able to calculate the derivative of the cost function in order to use gradient-based optimization methods. All commonly used cost functions are differentiable.
• Continuous: The function should be continuous for all possible predicted and actual output values.

Mean Squared Error Cost Function

The most commonly used cost function for linear regression is the mean squared error (MSE). As the name suggests, MSE calculates the average squared difference between the predicted and actual values.

Here is the equation for MSE:

$$J(β) = \frac{1}{2m} \sum_{i=1}^{m} (f(x^{(i)})−y^{(i)})^2$$

where:

• J is the cost function
• m is the number of training examples
• x^(i) and y^(i) represent the ith input and output values
• f(x^(i)) is the model‘s prediction for the ith input

Let‘s break this down step-by-step:

1. For each training example, calculate the squared difference between the model‘s prediction f(x^(i)) and the actual value y^(i)
2. Sum up the squared differences for all m training examples
3. Divide by 2m to get the average

The factor of 1/2 is included in the definition as a convenience for calculus – it makes the derivative simpler.

Calculating MSE: An Example

To make things concrete, let‘s walk through calculating the MSE for a simple linear regression problem. Suppose we have the following training data of experience (in years) and salary (in thousands):

Experience (x)	Salary (y)
1	40
2	50
3	60
4	70
5	80

We‘ll model salary as a linear function of experience:

Salary = β0 + β1 * Experience

Let‘s arbitrarily choose model parameters of β0 = 25 and β1 = 10. This means we are predicting a base salary of $25,000 plus an additional $10,000 per year of experience.

To calculate the MSE, we first get the model‘s predictions:

Experience (x)	Actual Salary (y)	Predicted Salary (f(x))	Squared Error
1	40	25 + 10*1 = 35	(40-35)^2 = 25
2	50	25 + 10*2 = 45	(50-45)^2 = 25
3	60	25 + 10*3 = 55	(60-55)^2 = 25
4	70	25 + 10*4 = 65	(70-65)^2 = 25
5	80	25 + 10*5 = 75	(80-75)^2 = 25

Summing the squared errors gives us 25 + 25 + 25 + 25 + 25 = 125. Plugging into the MSE formula with m=5 training examples:

MSE = 1/(25) 125 = 12.5

An MSE of 12.5 means that on average, this model‘s predictions are off by about $√12.5 ≈ $3,536. Obviously this particular choice of model parameters does not fit the data well. The goal is to find the values of β0 and β1 that minimize the MSE cost function.

Gradient Descent Optimization

One way to find the minimum of a cost function is to use gradient descent. Gradient descent is an iterative optimization algorithm that tweaks parameters in the direction that minimizes the cost.

Here‘s the basic procedure:

1. Initialize the model parameters (weights and bias) to some random values
2. Calculate the gradient (partial derivatives) of the cost function with respect to each parameter
3. Adjust each parameter by a small amount in the direction that reduces cost
4. Repeat steps 2-3 until cost converges to a minimum

The size of the adjustment in step 3 is determined by the learning rate hyperparameter. A smaller learning rate will take longer to converge but is less likely to overshoot the minimum.

There are several variants of gradient descent that differ in how much data is used to calculate the gradient at each iteration:

• Batch gradient descent uses the entire training set at each iteration. This gives the true gradient but is computationally expensive for large datasets.
• Stochastic gradient descent (SGD) uses a single randomly selected example at each iteration. SGD is faster and has a regularizing effect but can be noisy.
• Mini-batch gradient descent strikes a balance by using a small random subset of examples at each iteration.

Gradient descent is guaranteed to find the global minimum for convex cost functions like MSE. However, it can get stuck in local minima for non-convex cost functions. This is one reason convex cost functions are preferred.

Alternative Cost Functions

While MSE is the most commonly used cost function for linear regression, there are some alternatives that can be used in certain situations:

• Mean Absolute Error (MAE) calculates the average absolute difference between predicted and actual values. MAE is more robust to outliers than MSE but is not differentiable at 0, which can cause issues for gradient descent. MAE corresponds to L1 regularization.

• Huber loss is a combination of MSE and MAE that is quadratic for small errors and linear for large errors. This makes it more robust to outliers while still being differentiable. The cutoff between quadratic and linear is determined by a hyperparameter δ.

• Log-Cosh loss is another loss that is approximately quadratic for small errors and approximately linear for large errors, but is completely differentiable and avoids the hyperparameter of Huber loss.

Limitations and Considerations

Cost functions are a central component of training machine learning models but there are a few things to keep in mind:

• The choice of cost function should be aligned with the goal of the model. MSE and its relatives focus on getting the average prediction close to the average result, but may not be appropriate if we are more concerned with avoiding large errors.
• Adjusting the cost function is not the only way to improve a model. Using a more complex model, adding regularization, feature engineering, and collecting more data are other actions that can help.
• Minimizing cost on the training set does not guarantee good performance on new data. We need to use techniques like cross-validation to assess generalization and avoid overfitting to the training set.
• For large datasets and complex models, finding the exact minimum of the cost function may be computationally infeasible. Approximate methods like stochastic gradient descent are commonly used in practice.

Summary

Let‘s review the key points about using cost functions in linear regression:

• Cost functions quantify how well a model fits the training data by calculating the prediction error
• Mean squared error (MSE) is the most common cost function for linear regression
• Gradient descent is an optimization algorithm that finds the model parameters that minimize cost
• There are alternative cost functions like MAE and Huber loss that can be more appropriate in some cases
• Minimizing cost is necessary but not always sufficient for getting a model that generalizes well

I hope this gives you a solid conceptual understanding of cost functions and their role in linear regression. While the focus was on linear models, similar principles apply to cost functions for other types of models in machine learning. Being able to choose and interpret an appropriate cost function is a key skill in applying machine learning to real problems.