Building the Legendary VGG Net from Scratch with Python: An In-Depth Guide

Introduction

The VGG network, introduced by Simonyan and Zisserman in their seminal 2014 paper "Very Deep Convolutional Networks for Large-Scale Image Recognition", was a milestone in the evolution of convolutional neural networks (CNNs) for computer vision. In this comprehensive tutorial, we‘ll dive deep into the groundbreaking architecture, walking through a complete implementation from scratch in Python. Along the way, we‘ll explore the key design choices and historical context that made VGG net so impactful, as well as practical tips for training your own models. By the end, you‘ll have a solid grasp of both the theoretical underpinnings and hands-on application of this influential piece of deep learning history. Let‘s jump in!

The Need for Depth: Context and Motivation for VGG Net

To appreciate VGG net‘s significance, we need to understand the state of convolutional neural networks circa 2014. The dominant paradigm, exemplified by 2012‘s winning AlexNet architecture, was that increasing depth much beyond a half dozen or so layers was challenging and not necessarily beneficial. The prevailing wisdom held that deeper networks would be harder to train and more prone to overfitting.

VGG net‘s key insight was that these limitations could be overcome by a judicious, highly modularized architecture built on two key principles:

1. Use many layers of small (3×3) convolutional filters rather than fewer layers of larger (5×5 or 7×7) filters. This allows the network to learn more complex, discriminative features while keeping computational cost and number of parameters manageable.

2. Steadily decrease spatial resolution with max pooling as feature maps get deeper. This provides translation invariance and keeps computations tractable.

With this elegant recipe, VGG net showed that much deeper networks (16-19 layers) could indeed achieve state-of-the-art performance on large-scale image recognition tasks, provided sufficient training data and regularization. This was a pivotal result that helped spark the "very deep learning" revolution in the years that followed.

Under the Hood: VGG Net Architecture

Let‘s now take a closer look under the hood at VGG net‘s architecture. We‘ll focus on the 16-layer variant (VGG-16) as it offers a good balance of depth and complexity. Here‘s a layer-by-layer breakdown:

A few salient points about the architecture:

• All conv layers use small 3×3 filters with stride 1 and padding to preserve spatial dimensions. This allows the network to learn spatial hierarchies of features efficiently.
• Max pooling is used to reduce spatial dimensions after certain conv layers. This provides translation invariance and keeps computation manageable as depth increases.
• Rectified linear unit (ReLU) activations are used throughout for nonlinearity, except for the final softmax output layer.
• The final three fully connected (FC) layers provide high-level reasoning and aggregation of features for classification. The last layer outputs a probability distribution over the 1000 ImageNet classes.

Notably, the original VGG net did not employ batch normalization or dropout – two regularization techniques that have since become standard in deep CNNs. This is a testament to the robustness of the core architecture.

Implementing VGG-16 from Scratch in PyTorch

Armed with an understanding of VGG net‘s architecture, we‘re ready to implement it in code. We‘ll be using PyTorch, but the same concepts apply in any modern deep learning framework. Let‘s start by defining the model architecture in a reusable VGGNet class:

import torch
import torch.nn as nn

VGG_types = {
    ‘VGG11‘: [64, ‘M‘, 128, ‘M‘, 256, 256, ‘M‘, 512, 512, ‘M‘, 512, 512, ‘M‘],
    ‘VGG13‘: [64, 64, ‘M‘, 128, 128, ‘M‘, 256, 256, ‘M‘, 512, 512, ‘M‘, 512, 512, ‘M‘],
    ‘VGG16‘: [64, 64, ‘M‘, 128, 128, ‘M‘, 256, 256, 256, ‘M‘, 512, 512, 512, ‘M‘, 512, 512, 512, ‘M‘],
    ‘VGG19‘: [64, 64, ‘M‘, 128, 128, ‘M‘, 256, 256, 256, 256, ‘M‘, 512, 512, 512, 512, ‘M‘, 512, 512, 512, 512, ‘M‘],
}

class VGGNet(nn.Module):
    def __init__(self, in_channels=3, num_classes=1000, architecture=‘VGG16‘):
        super(VGGNet, self).__init__()
        self.in_channels = in_channels
        self.conv_layers = self.create_conv_layers(VGG_types[architecture])

        self.fcs = nn.Sequential(
            nn.Linear(512 * 7 * 7, 4096),
            nn.ReLU(),
            nn.Dropout(p = 0.5),
            nn.Linear(4096, 4096),
            nn.ReLU(),
            nn.Dropout(p = 0.5),
            nn.Linear(4096, num_classes)
        )

    def forward(self, x):
        x = self.conv_layers(x)
        x = x.reshape(x.shape[0], -1)
        x = self.fcs(x)
        return x

    def create_conv_layers(self, architecture):
        layers = []
        in_channels = self.in_channels

        for x in architecture:
            if type(x) == int:
                out_channels = x
                layers += [
                    nn.Conv2d(
                        in_channels=in_channels,
                        out_channels=out_channels,
                        kernel_size=(3,3),
                        stride=(1,1),
                        padding=(1,1)
                    ),
                    nn.ReLU(),
                ]
                in_channels = x
            elif x == ‘M‘:
                layers += [nn.MaxPool2d(kernel_size=(2,2), stride=(2,2))]

        return nn.Sequential(*layers)

Here‘s a step-by-step breakdown:

1. We define a dictionary VGG_types that maps the names of different VGG variants to their conv layer specifications. ‘M‘ denotes a max pooling layer.

2. The VGGNet class constructor takes the input channels, number of classes, and architecture name (defaulting to ‘VGG16‘).

3. The create_conv_layers method constructs the conv layers according to the provided architecture spec, using 3×3 conv filters with ReLU activations and 2×2 max pooling.

4. The fully connected layers are defined in the constructor using nn.Sequential. This includes dropout for regularization.

5. The forward method passes the input through the conv layers, reshapes it, and passes it through the FC layers to get the output class scores.

We can instantiate a VGG-16 model for ImageNet classification with just a few lines:

device = ‘cuda‘ if torch.cuda.is_available() else ‘cpu‘ 
model = VGGNet(in_channels=3, num_classes=1000, architecture=‘VGG16‘).to(device)

Training the Model

With our VGG-16 model implemented, the next step is to train it on data. The original VGG net was trained on the large-scale ImageNet dataset (1.2 million images across 1000 classes), but for demonstration purposes we could use a smaller dataset like CIFAR-100 (60,000 32×32 images across 100 classes).

A typical training loop in PyTorch looks like this:

criterion = nn.CrossEntropyLoss() 
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

num_epochs = 100
for epoch in range(num_epochs):
    for images, labels in train_loader:
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward pass and optimize
        optimizer.zero_grad()
        loss.backward()        
        optimizer.step()

Some key points about the training process:

• We use cross-entropy loss as the objective function for this multi-class classification problem.
• Stochastic gradient descent (SGD) with momentum is the optimizer used in the original VGG net paper. The learning rate and momentum are hyperparameters that can be tuned.
• The training loop iterates over the dataset for a number of epochs, with each epoch consisting of multiple batches.
• For each batch, we move the data to the GPU (if available), do a forward pass through the model to get the predicted outputs, compute the loss, and then do a backward pass to compute gradients and update the weights.
• It‘s important to zero the gradients with optimizer.zero_grad() before each backward pass, as PyTorch accumulates gradients by default.

Of course, we‘d also want to periodically evaluate the model on a validation set during training to monitor progress and catch overfitting. After training for enough epochs (which can take days for a dataset like ImageNet!), we‘d finally evaluate performance on a held-out test set.

Some additional tips and best practices for training VGG net:

• Use data augmentation techniques like random cropping, flipping, and color jittering to improve robustness and reduce overfitting. The original VGG net paper employed extensive data augmentation.
• Normalize the input data by subtracting the per-channel mean (computed on the training set). This helps the optimization process.
• Start with a higher learning rate and decrease it by a factor of 10 a few times during training as the validation accuracy plateaus.
• Use weight decay (L2 regularization) of 5e-4 or similar to penalize large weights and combat overfitting.
• Consider using learning rate scheduling techniques like cosine annealing to adjust the learning rate smoothly over the course of training.

With a well-designed training pipeline and sufficient compute power, our from-scratch VGG-16 model has the potential to achieve strong results on a variety of image classification tasks. The original VGG-16 achieved 92.7% top-5 test accuracy on ImageNet – a result that, while since surpassed, remains impressive to this day.

Applications and Legacy

The impact of VGG net extends far beyond its performance on ImageNet. The weights learned by training on such a large, diverse dataset captured a rich hierarchy of visual features that proved highly transferable to other computer vision tasks.

In particular, using a pre-trained VGG net as a feature extractor became a go-to technique for tasks like object detection, semantic segmentation, and style transfer. The activations from different layers of the network could be used as off-the-shelf descriptors for images, often outperforming hand-engineered features.

This transferability of deep features, which VGG net helped popularize, remains a cornerstone of modern computer vision. It‘s what allows us to achieve strong results on tasks with limited labeled data by leveraging the visual knowledge learned on large datasets like ImageNet.

While VGG net has since been outperformed by more modern architectures like ResNet and EfficientNet, its elegance and simplicity continue to make it a popular choice for teaching and learning purposes. The VGG paper is one of the most highly-cited in the field of artificial intelligence (over 80,000 citations as of 2023) and its insights have shaped the evolution of CNN architectures ever since.

Conclusion

In this deep dive, we‘ve explored the groundbreaking VGG network from concept to code. We‘ve walked through its key architectural innovations, historical context, and practical implementation details.

We‘ve seen how VGG net demonstrated the power of very deep convolutional networks and helped usher in a new era of visual recognition powered by hierarchical feature learning. Its elegant, modular design continues to be highly influential and its pre-trained weights remain a valuable resource for practitioners.

Implementing VGG net from scratch is a great way to solidify your understanding of the fundamentals of deep learning for computer vision. By digging into the details – the tensor shapes, architectural choices, training process – you develop intuitions that will serve you well as you explore more advanced concepts and architectures.

Of course, this tutorial only scratches the surface of what‘s possible with VGG net and CNNs in general. I encourage you to experiment with different datasets, hyperparameters, and modifications to the architecture. You might be surprised at what insights and results you uncover!

I hope this guide has given you a solid foundation for your deep learning journey. The complete code for the tutorial is available on Github. If you have any questions or thoughts, feel free to reach out. Until next time, happy coding!

References

1. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
2. PyTorch documentation: https://pytorch.org/docs/stable/index.html
3. ImageNet dataset: https://www.image-net.org/
4. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
5. Tan, M., & Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning (pp. 6105-6114). PMLR.