Build Your First Artificial Neural Network Using PyTorch

Artificial neural networks (ANNs) are a powerful machine learning technique inspired by the structure and function of biological neural networks in the brain. Just as the brain uses a dense web of interconnected neurons to process sensory inputs, learn patterns, and control behaviors, ANNs utilize artificial neurons and weighted connections to perform tasks like classification, prediction, pattern recognition, and more.

In recent years, deep learning with ANNs has achieved breakthrough results in domains like computer vision, natural language processing, speech recognition, and robotics. Tools like PyTorch have made it easier than ever for practitioners to experiment with neural networks and harness their power.

In this tutorial, we‘ll walk through the process of building and training a simple ANN using PyTorch. By the end, you‘ll understand the key components and mechanisms of neural networks and be able to start applying them to your own datasets and problems. Let‘s dive in!

Neural Network Basics

At its core, an artificial neural network consists of:

• Neurons (nodes): The basic processing units, loosely analogous to neurons in the brain. Each neuron receives weighted inputs, applies an activation function, and produces an output.

• Layers: Neurons are organized into layers – an input layer that receives the data, one or more hidden layers that learn representations, and an output layer that makes predictions. Information flows through the layers in one direction (feedforward).

• Weights: The connections between neurons have associated weights that determine the strength of influence of one neuron on another. These weights are learnable parameters that are adjusted via training.

• Activation functions: Each neuron applies a nonlinear activation function to its weighted inputs to produce an output. Popular choices include sigmoid, tanh, ReLU, and variants. Activation functions enable the network to learn complex patterns and relationships.

• Loss function: Measures how well the network‘s predictions match the expected outputs. The goal of training is to minimize this loss.

• Optimizer: The algorithm used to update the weights in the direction that reduces the loss. Backpropagation with gradient descent is the standard approach.

During training, the network is shown many example inputs and iteratively adjusts its weights to map those inputs to the correct outputs. This enables the network to learn the underlying patterns and generalize to new, unseen data.

The universal approximation theorem states that a feedforward neural network with at least one hidden layer can approximate any continuous function to an arbitrary degree of accuracy, given enough hidden units. This means neural networks are powerful, flexible learners.

Why PyTorch?

PyTorch is an open-source machine learning library developed primarily by Facebook‘s AI Research lab. It has seen rapid adoption in the research community and industry due to its dynamic computation graphs, imperative style, and easy-to-use APIs.

Some key features and benefits of PyTorch include:

• Dynamic computation graphs: PyTorch uses dynamic computation graphs that can be defined and manipulated on the fly. This provides a more intuitive and flexible interface compared to frameworks that use static graphs.

• Imperative style: PyTorch code is executed eagerly, making debugging easier. You can use standard Python statements, set breakpoints, and inspect variables at runtime.

• Pythonic: PyTorch meshes well with the rest of the Python data science stack and feels native to Python developers. It can be easily integrated with popular libraries like NumPy.

• Customizable: All the building blocks for deep learning are available as simple, well-documented APIs. This makes it easy to experiment and customize neural network architectures and training procedures.

• Performance: PyTorch is fast and memory efficient, thanks to its C++ backend, built-in support for hardware acceleration, and asynchronous execution. It can handle large-scale production workloads.

• Community: PyTorch has a thriving open source community with strong momentum and rapid development. The PyTorch ecosystem includes a variety of domain libraries and extensions.

In short, PyTorch provides a best-of-both-worlds combination of high-level APIs for neural networks and deep integration with the Python ecosystem. It‘s an excellent choice whether you‘re a beginner learning the concepts or an advanced practitioner needing flexibility and performance.

Building a Feedforward Neural Network

Now let‘s get hands-on and build a simple feedforward neural network to classify handwritten digits from the famous MNIST dataset. We‘ll use PyTorch to define the network architecture, load the data, train the model, and evaluate its performance.

Step 1: Install PyTorch and Prepare Data

If you don‘t have PyTorch installed already, you can do so via pip:

!pip install torch torchvision

PyTorch provides convenient APIs for downloading and preparing common datasets. Let‘s load the MNIST training and test data:

import torch  
from torchvision import datasets
from torchvision.transforms import ToTensor

training_data = datasets.MNIST(
    root="data",
    train=True,
    download=True,
    transform=ToTensor(),
)

test_data = datasets.MNIST(
    root="data",
    train=False,
    download=True,
    transform=ToTensor(),
)

This downloads the MNIST dataset to a local data directory and wraps it in PyTorch Dataset objects for convenient access. The ToTensor transform normalizes the images.

Next we can create PyTorch DataLoader objects to manage batching and shuffling of the data during training:

from torch.utils.data import DataLoader

batch_size = 64
dataloader = DataLoader(training_data, batch_size=batch_size)
test_dataloader = DataLoader(test_data, batch_size=batch_size)

The DataLoader will give us batches of size 64 on each iteration, drawn randomly from the dataset.

Step 2: Define the Neural Network

Now we can define our neural network architecture by subclassing the torch.nn.Module base class:

import torch.nn as nn

class FeedforwardNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten()
        self.dense_layers = nn.Sequential(
            nn.Linear(28*28, 128), 
            nn.ReLU(),
            nn.Linear(128, 10)
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.dense_layers(x)
        return logits

Our FeedforwardNet class defines a simple network with the following components:

• A flatten layer to reshape the 2D input images into 1D vectors
• A dense (fully connected) layer with 128 units and ReLU activation
• A final dense layer with 10 units (one per class) to output logits

The forward method is called when the model is invoked on input data. It passes the data through the network layers and returns the output logits.

We can create an instance of our model and inspect it:

model = FeedforwardNet()
print(model)

This prints a summary of the model showing the type and size of each layer.

Step 3: Train the Model

To prepare the model for training, we need to:

• Create an optimizer to update the weights
• Define a loss function to quantify the error
• Decide for how many epochs to train

A common choice is to use the Adam optimizer with default parameters and cross-entropy loss. We can train for 10 epochs to start:

from torch import optim

loss_fn = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)
epochs = 10

Now we can write a training loop that:

1. Iterates over the dataset for the specified number of epochs
2. Breaks the data into batches using the DataLoader
3. Performs a forward pass through the model for each batch
4. Computes the loss by comparing the outputs to the labels
5. Performs a backward pass to compute gradients
6. Steps the optimizer to update the model weights
7. Resets the gradients for the next iteration
8. Prints the training loss every 1000 batches

Here‘s the code:

for epoch in range(epochs):
    print(f"Epoch {epoch+1}\n-------------------------------")
    for batch_idx, (x, y) in enumerate(dataloader):

        # Forward pass
        logits = model(x)
        loss = loss_fn(logits, y)

        # Backpropagation
        optimizer.zero_grad() 
        loss.backward()
        optimizer.step()

        if batch_idx % 1000 == 0:
            print(f"Epoch: {epoch+1} | Batch: {batch_idx:5d} | Loss: {loss:.4f}")

As the model trains, you should see the training loss decrease over time as the network learns to map the image pixels to the correct digit labels.

Step 4: Evaluate Performance

Once the model is trained, we can check its performance on the test set, which it hasn‘t seen during training. We‘ll use the trained model to make predictions on each test batch and calculate the fraction of correct predictions:

model.eval()
with torch.no_grad():
    correct = 0
    total = 0
    for x, y in test_dataloader:
        logits = model(x)
        predictions = torch.argmax(logits, dim=1)
        total += y.size(0)
        correct += (predictions == y).sum()

    print(f‘Test accuracy: {100*(correct/total):.2f}%‘)

The model‘s accuracy on the test set is a good estimate of how well it generalizes. A well-trained feedforward network can achieve over 98% accuracy on MNIST.

Next Steps

Congratulations – you‘ve trained your first neural network in PyTorch! There are many ways you can build on this example:

• Experiment with adding more layers to the network and see how this affects training speed and accuracy
• Try different activation functions like tanh or Leaky ReLU
• Implement a convolutional neural network (CNN) to capture spatial patterns in the images
• Apply regularization techniques like dropout or weight decay to reduce overfitting
• Visualize the learned weights and activations to gain insight into what the network is learning
• Train on other datasets like CIFAR-10 or Fashion MNIST

PyTorch supports all the common neural network architectures, including:

• Convolutional Neural Networks (CNNs) for images and video
• Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTMs) for sequences and time series
• Transformers and attention-based models for natural language processing
• Autoencoders and Variational Autoencoders (VAEs) for representation learning and generative modeling
• Generative Adversarial Networks (GANs) for synthesizing new data

The PyTorch documentation is a great resource for learning about the features and APIs available. The PyTorch developer ecosystem is also very active on forums like Discuss PyTorch.

Conclusion

We‘ve covered a lot of ground in this post, including:

• The basic concepts and components of artificial neural networks
• An overview of PyTorch and its key features
• Step-by-step implementation of a feedforward neural network for classifying MNIST digits
• The PyTorch workflows for preparing data, defining models, and training
• Evaluating model performance on a held-out test set
• Ideas for extending and enhancing the basic model

With its elegant design and flexibility, PyTorch is a top choice for both learning the fundamentals of deep learning and implementing state-of-the-art models. The skills you‘ve learned here will serve as a foundation as you continue your deep learning journey.

I encourage you to keep exploring and experimenting with neural networks. Seemingly simple models can exhibit surprising intelligence and capability. Best of luck!