Gradient Descent: Design Your First Machine Learning Model

Introduction

Gradient descent is an optimization algorithm that forms the backbone of modern machine learning. It is the workhorse behind training powerful models like deep neural networks on large datasets. But what exactly is gradient descent and how does it work under the hood?

In this tutorial, we‘ll demystify gradient descent from the ground up. You‘ll learn what gradient descent is conceptually, how to implement it from scratch in Python, and how to apply it to train your first machine learning model. Whether you‘re new to ML or need a refresher on this core concept, this post will equip you with an intuitive understanding of this all-important algorithm. Let‘s dive in!

The Gradient Descent Algorithm

At its core, gradient descent is an iterative algorithm to minimize a cost function J(θ) parameterized by a model‘s parameters θ. The goal is to find the optimal parameters θ that minimize the cost function, thereby fitting the model to the training data.

Intuitively, it works by iteratively taking steps in the direction of steepest descent of the cost function, until it reaches a local minimum. Mathematically, this is achieved by updating the parameters in the opposite direction of the gradient of the cost function ∇J(θ):

θ := θ – α ∇J(θ)

Here α is the learning rate that controls the size of the steps we take in the gradient descent direction. The gradient ∇J(θ) is a vector of partial derivatives ∂J/∂θ, pointing in the direction of steepest ascent. By iteratively updating θ to move in the opposite direction, gradient descent localizes lower values of J(θ) until it converges to a minimum.

Implementing Gradient Descent in Python

Now that you have the conceptual idea, let‘s implement gradient descent in Python and apply it to a toy problem. We‘ll use PyTorch, but the concepts readily apply to any deep learning framework.

First, let‘s generate a small synthetic regression dataset:

import torch
import matplotlib.pyplot as plt

# Generate synthetic data
torch.manual_seed(42)
X = torch.rand(100, 1)
y = 1 + 2 * X + 0.1*torch.randn(X.shape)

We have 100 data points generated from a noisy linear model y = 1 + 2x. Our task will be to recover the parameters of this line using gradient descent.

Next, let‘s define our model and the mean squared error loss function:

# Linear regression model
def model(x, w, b):
    return w*x + b

# MSE loss
def mse(t1, t2):
    diff = t1 - t2
    return torch.sum(diff ** 2) / diff.numel()  

The model is just a linear function with slope w and intercept b. The MSE loss measures the average squared difference between predictions and targets.

Now let‘s initialize the parameters and set up the training loop with gradient descent:

# Initialize parameters 
w = torch.randn(1, requires_grad=True)
b = torch.randn(1, requires_grad=True)
print(f‘Initial parameters: w={w.item():.3f}, b={b.item():.3f}‘)

# Hyperparameters
lr = 0.1
n_epochs = 100

# Train with gradient descent
for epoch in range(n_epochs):

    # Forward pass
    y_pred = model(X, w, b) 
    loss = mse(y_pred, y)

    # Backward pass
    loss.backward()

    # Update parameters  
    with torch.no_grad():
        w -= w.grad * lr
        b -= b.grad * lr
        w.grad.zero_()
        b.grad.zero_()

    # Print progress
    if (epoch+1) % 10 == 0:
        print(f‘Epoch [{epoch+1}/{n_epochs}], Loss: {loss.item():.4f}‘)

print(f‘\nFinal parameters: w={w.item():.3f}, b={b.item():.3f}‘)

Let‘s break this down:

1. We initialize the parameters w and b randomly, and mark them for gradient computation with requires_grad.
2. In each epoch, we perform a forward pass to compute the predictions and loss.
3. We then perform a backward pass with loss.backward() to compute gradients of the loss w.r.t the parameters.
4. Finally, we update the parameters by taking a step of size lr in the negative gradient direction. Crucially, we zero the gradients afterwards so they don‘t accumulate across epochs.

After 100 epochs of gradient descent, we arrive at final parameters very close to the ground truth w=2 and b=1 that we used to generate the data! Here‘s how the model‘s fit evolves during training:

w_hist, b_hist = [], []

w = torch.randn(1, requires_grad=True) 
b = torch.randn(1, requires_grad=True)

for epoch in range(n_epochs):
    y_pred = model(X, w, b)
    loss = mse(y_pred, y) 
    loss.backward()

    with torch.no_grad():
        w -= w.grad * lr
        b -= b.grad * lr
        w.grad.zero_() 
        b.grad.zero_()

    w_hist.append(w.item())
    b_hist.append(b.item())

plt.figure(figsize=(14,10))
plt.subplot(221)
plt.plot(w_hist, lw=3)
plt.plot(b_hist, lw=3)
plt.legend([‘w‘, ‘b‘])
plt.xlabel(‘Epochs‘)
plt.ylabel(‘Parameters‘)

for i, epoch in enumerate([0, 9, 99]):
    plt.subplot(2,2,i+2)
    plt.scatter(X, y)
    plt.plot(X, w_hist[epoch] * X + b_hist[epoch], color=‘red‘)
    plt.title(f‘Epoch {epoch+1}‘)

plt.tight_layout()
plt.show()

This visualizes the evolution of the model‘s predictions as the parameters are updated by gradient descent. Initially, the predictions are poor with the line far from the data. But as gradient descent proceeds, the model gradually converges to a good fit. We can see the parameters w and b progressively approach their true values.

Tips and Tricks

Gradient descent may seem straightforward, but achieving stable and efficient optimization often takes some tuning. Here are some key tips:

• Learning rate: This is the most important hyperparameter. Set it too low and training will progress slowly. Set it too high and the loss may diverge to infinity. Monitor the loss during training, and adjust lr as needed. Typical values range from 0.1 to 1e-6.

• Initialization: The starting point can determine whether gradient descent converges at all. Initialize weights to small random values to break symmetry. For deep networks, techniques like Xavier initialization can help signals propagate.

• Shuffling: For stochastic/mini-batch GD, shuffle the training data every epoch to reduce variance and avoid cycles. For similar reasons, avoid using a constant learning rate.

• Normalize inputs: Scale input data to have zero mean and unit variance. This helps gradient descent converge more stably and quickly as features are on similar scales.

• Regularization: Add regularization terms to the loss like L1/L2 penalties on weights to combat overfitting, or use techniques like dropout.

• Visualize: Plot metrics like the loss, gradients, parameter distributions during training. Visualizing the model‘s predictions and studying its errors offers invaluable debugging information.

Conclusion

In this post, we developed an intuitive understanding of the gradient descent algorithm and showed how to apply it to train a simple ML model. We started with the conceptual formulation of gradient descent and its update rule. We then generated a toy 1D regression problem and used gradient descent to learn the parameters of a linear model to fit the data.

Through visualizations, we saw how the model‘s predictions evolved as its parameters were incrementally updated by gradient descent. We also discussed tips and tricks to make gradient descent run more smoothly, like tuning the learning rate and monitoring training dynamics.

Needless to say, we‘ve only scratched the surface of this vast topic. There are more sophisticated variants of gradient descent like SGD, Adam, RMSProp, etc. that are better suited for optimizing complex models on large datasets. Techniques like learning rate scheduling, momentum, and adaptive methods can also help accelerate and stabilize convergence.

Nonetheless, the core concepts remain the same, and with this foundation you‘re well equipped to start training your own models! I encourage you to re-implement this toy example, and play with the model architecture and hyperparameters to build intuition. Then try your hand at applying gradient descent to more complex models and datasets.

I hope this post demystified gradient descent and gave you a practical guide for getting started with it. The complete code for this tutorial is available here. Drop any questions in the comments below, and happy descending!