Getting Started with Machine Learning: Implementing Linear Regression from Scratch in Python

Introduction

Linear regression is one of the most fundamental and widely-used machine learning algorithms. It models the relationship between variables by fitting a linear equation to observed data. Linear regression has many practical applications, such as predicting sales, analyzing trends, or estimating costs.

While powerful libraries like sci-kit learn make it easy to implement linear regression in a few lines of code, building it from scratch is a valuable exercise to truly understand how it works under the hood. In this post, we‘ll walk through the process step-by-step in Python. By the end, you‘ll be able to apply linear regression to your own datasets and have a strong foundation to explore more advanced machine learning concepts.

Mathematical Intuition

Before we dive into the code, let‘s establish some mathematical intuition. The goal of linear regression is to find the line of best fit through a set of data points. We can represent this line with the equation:

y = mx + b

Where y is the predicted value, m is the slope of the line, x is the input value, and b is the y-intercept.

To quantify how well a given line fits the data, we define a cost function J. A common choice is the mean squared error (MSE):

J = (1/N) * Σ(yᵢ – ŷᵢ)²

This sums the squared differences between the actual y values and the predicted y values (denoted ŷ), giving us the average squared error.

Our objective is to minimize the cost function by adjusting the parameters m and b. We can do this using an optimization algorithm called gradient descent. The general idea is to start with random values for m and b, then iteratively update them in the direction that minimizes J.

The gradient descent update rule is:

m := m – α ∂J/∂m
b := b – α ∂J/∂b

Where α is the learning rate that controls the size of the update steps. The partial derivatives ∂J/∂m and ∂J/∂b tell us the slope of the cost function with respect to m and b.

With the key mathematical components defined, let‘s see how to translate this into code.

Implementing in Python

We‘ll start by importing the necessary libraries. NumPy will allow us to efficiently perform mathematical operations on arrays, and Matplotlib will help visualize the results.

import numpy as np
import matplotlib.pyplot as plt

Next, let‘s assume we already have a dataset of x and y values stored in two Python lists called X and y. We‘ll convert these to NumPy arrays to take advantage of vectorized operations:

X = np.array(X)
y = np.array(y)

With the data prepared, we can start defining the key components of our linear regression implementation.

First is the model itself, which will take in the current parameter values and an input x value and return the predicted y value:

def model(X, m, b):
    return m*X + b

Next is the cost function, which will take in the current parameter values and the full dataset and return the average squared error:

def cost(X, y, m, b):
    y_pred = model(X, m, b)
    return np.mean((y - y_pred)**2)

The gradient descent function will perform a single optimization step by updating the parameters m and b in the direction of the negative gradient:

def gradient_descent(X, y, m, b, alpha):
    y_pred = model(X, m, b)
    m = m - alpha * (1/len(X)) * np.sum(2*(y_pred-y)*X)
    b = b - alpha * (1/len(X)) * np.sum(2*(y_pred-y))
    return m, b  

To train the model, we‘ll loop through the gradient descent update step for a set number of iterations, or until the cost falls below a certain threshold:

m = 0
b = 0
alpha = 0.01
num_iterations = 1000

for i in range(num_iterations):
    m, b = gradient_descent(X, y, m, b, alpha)
    if i % 100 == 0:
        print(f"Iteration {i}: Cost = {cost(X, y, m, b):.4f}")

print(f"Final model: y = {m:.4f}x + {b:.4f}")

This will print out the cost every 100 iterations so we can monitor progress, as well as the final values of m and b that define our line of best fit.

To make predictions on new, unseen data points, we simply plug them into our model function with the learned parameters:

X_new = np.array([8, 12, 15])
y_pred = model(X_new, m, b)
print(f"Predicted y values: {y_pred}")  

Evaluating Performance

To quantify how well our model fits the data, we can calculate some evaluation metrics on the training data:

y_pred = model(X, m, b)
mse = np.mean((y - y_pred)**2)
rmse = np.sqrt(mse)
print(f"MSE: {mse:.4f}")
print(f"RMSE: {rmse:.4f}") 

The mean squared error (MSE) tells us the average squared difference between the actual and predicted y values. Taking the square root gives us the root mean squared error (RMSE), which is in the same units as the original y values and thus more interpretable. The lower the MSE and RMSE, the better the model fits the data.

Another useful metric is the coefficient of determination, or R-squared. This represents the proportion of variance in the dependent variable that is predictable from the independent variable. An R-squared of 1 indicates that the regression predictions perfectly fit the data.

ss_tot = np.sum((y - np.mean(y))**2)
ss_res = np.sum((y - y_pred)**2)
r_squared = 1 - (ss_res / ss_tot)
print(f"R-squared: {r_squared:.4f}")

Visualizing Results

To visualize how well the regression line fits the actual data points, we can use Matplotlib to create a scatter plot of the x and y values, and plot the line of best fit over top:

plt.figure(figsize=(8,5))
plt.scatter(X, y, label=‘Data Points‘)
plt.plot(X, model(X, m, b), c=‘r‘, label=‘Line of Best Fit‘)
plt.xlabel(‘X‘)
plt.ylabel(‘y‘) 
plt.title(‘Linear Regression Results‘)
plt.legend()
plt.show()

This provides a nice visual confirmation of the model‘s performance to complement the quantitative metrics.

Comparing to Sci-kit Learn

As mentioned earlier, libraries like sci-kit learn make it very easy to implement linear regression in practice. We can compare the results of our from-scratch implementation to sci-kit learn‘s:

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X.reshape(-1,1), y)

print(f"Slope: {model.coef_[0]:.4f}")  
print(f"Intercept: {model.intercept_:.4f}")

y_pred = model.predict(X.reshape(-1,1))
print(f"R-squared: {model.score(X.reshape(-1,1), y):.4f}")

In most cases, the results should be quite close (up to small differences due to the specific optimization algorithms used).

However, although our implementation is great for learning purposes, libraries like sci-kit learn offer many additional features, such as built-in regularization, performance enhancements, and integrations with other parts of the machine learning workflow. They have also been rigorously tested and optimized. For most real-world applications, it‘s recommended to leverage these established tools.

Potential Improvements

There are many ways we could expand on our basic linear regression implementation:

• Add regularization (e.g. L1/L2) to handle overfitting and improve generalization
• Experiment with other optimization algorithms like stochastic gradient descent or Adam
• Implement polynomial regression to fit non-linear relationships
• Extend to multiple linear regression to handle multiple input variables
• Explore other model evaluation techniques like k-fold cross validation

These would be great next steps to deepen your understanding and build even more powerful models.

Conclusion

We‘ve covered a lot of ground in this post, starting with the fundamental concepts of linear regression and building up to a complete working implementation in Python.

Some key takeaways:

• Linear regression is a powerful yet simple machine learning algorithm for modeling linear relationships between variables
• Gradient descent is an iterative optimization algorithm that minimizes the cost function to find the line of best fit
• Python libraries like NumPy and Matplotlib enable efficient computation and easy visualization
• Implementing algorithms from scratch is a great way to understand them at a deeper level
• Mean squared error, root mean squared error, and R-squared are useful metrics for evaluating regression models
• While from-scratch implementations are valuable for learning, libraries like sci-kit learn offer many additional features and optimizations that make them preferable for real-world use cases

I hope this post has given you a solid foundation for understanding and applying linear regression in practice. Feel free to use the code provided as a starting point for your own experiments and extensions. Happy learning!