Deep Residual Learning for Image Recognition (ResNet Explained)

Introduction

In recent years, deep learning has achieved remarkable breakthroughs in a variety of domains, especially computer vision. Convolutional neural networks (CNNs) have become the dominant approach for tasks like image classification, object detection, and semantic segmentation.

One of the key factors driving progress has been the ability to train increasingly deep networks with tens or even hundreds of layers. In theory, deeper networks have greater representational power and should lead to better performance. However, in practice, a phenomenon known as the vanishing gradient problem has limited the depth of trainable networks.

In 2015, researchers at Microsoft introduced a CNN architecture called the Deep Residual Network (ResNet) that allowed networks with over 100 layers to be effectively trained. ResNet utilizes skip connections to propagate gradients throughout the depth of the network, alleviating the vanishing gradient issue. The seminal paper "Deep Residual Learning for Image Recognition" ushered in a new era of deep learning for computer vision.

In this blog post, we‘ll take a closer look at the vanishing gradient problem, the key innovations of the ResNet architecture, and the impact ResNet has had on the field. Let‘s dive in!

The Vanishing Gradient Problem in Deep Networks

To understand the significance of ResNet, it‘s important to first examine the vanishing gradient problem that has historically plagued deep neural networks. During training, networks learn by backpropagating error gradients from the output layer all the way to the input layer, adjusting weights along the way to minimize the error.

However, as the number of layers increases, the gradients tend to get smaller and smaller as they are propagated backwards. This is because the gradients are multiplied by values less than one at each layer. After enough layers, the gradients essentially shrink to zero, making it very difficult to update the weights of the early layers in the network.

This is problematic because the earliest layers of CNNs tend to learn fundamental low-level features like edges and textures that are critical for the network‘s overall performance. If these layers can‘t be trained effectively, the full potential of deep networks can‘t be realized.

Various solutions had been proposed to mitigate vanishing gradients, such as careful initialization schemes and intermediate normalization layers. However, the problem still tended to crop up in very deep networks, stymieing progress.

The ResNet authors noted that simply stacking more layers on top of a baseline 18-layer CNN actually led to higher training error, even though the added layers gave the network more parameters and greater representational power. This indicates that the degradation is not caused by overfitting, but by an inability to train the deeper network to fit the training data in the first place. In other words – the vanishing gradients strike again!

Clearly, a new approach would be needed to harness the power of very deep networks. Enter the ResNet architecture.

The ResNet Architecture

Motivation

The key insight behind ResNets is that instead of expecting each layer to directly fit a desired underlying mapping H(x), it is easier to fit the residual F(x) = H(x) – x. The original mapping is then obtained by adding the identity x back to the residual: H(x) = F(x) + x.

Intuitively, if the optimal function is closer to the identity function than to a zero mapping, it should be easier for the layers to learn the residual. In the extreme case where the optimal H(x) is exactly the identity, it would be easy for the layers to drive the residual to zero rather than fit the identity.

Residual Block Formulation

This idea is implemented using residual blocks, which form the basic building block of ResNet architectures. Here is the mathematical formulation:

y = F(x, {Wi}) + x

where:

• x and y are the input and output of the block
• F(x, {Wi}) represents the residual mapping to be learned
• F + x is performed by a shortcut/skip connection

The residual function F is typically implemented by a few (usually 2 or 3) stacked convolutional layers. Each conv layer is typically followed by batch normalization and a ReLU activation.

The shortcut connection skips these layers and adds the identity x back to the output. When the dimensions of x and F don‘t match (e.g. at transition layers that halve the spatial resolution), a linear projection can be used to match the dimensions:

y = F(x, {Wi}) + Wsx

where Ws is a matrix that performs the linear projection. In most cases, the shortcuts are simple identity connections and Ws is not used.

Network Architectures

Using residual blocks as the fundamental building block, the ResNet paper introduced several CNN architectures for image classification:

• ResNet-18: 18 layer network with basic residual blocks
• ResNet-34: 34 layer network with basic residual blocks
• ResNet-50/101/152: 50/101/152 layer networks with bottleneck residual blocks

The bottleneck blocks used in the deeper networks are more computationally efficient. They use a stack of 1×1, 3×3, 1×1 conv layers where the 1×1 layers reduce and restore dimensions and the 3×3 layer operates on smaller input/output dimensions.

Experiments

Datasets and Training Details

The ResNet models were evaluated on two major image classification benchmarks:

1. ImageNet: A large dataset containing 1.28 million training images and 50k validation images across 1000 classes. Performance is typically measured by top-1 and top-5 error rates.
2. CIFAR-10: A smaller dataset with 50,000 training and 10,000 test 32×32 color images across 10 classes.

On ImageNet, the models were trained using SGD with momentum for 60-120 epochs with batch sizes of 256. Weights were initialized using a scheme designed for deep networks. On CIFAR, the models were trained for 200 epochs with batch size 128.

Results

The ResNet paper presented convincing empirical results demonstrating the effectiveness of residual learning for training deep networks:

1. Significant reduction in training and validation error for deep ResNets compared to plain counterpart networks, especially for depth beyond 20 layers. For example, a 34 layer ResNet had roughly half the validation error of an 18 layer plain net.

2. Ability to train extremely deep nets with 50, 101, and even 152 layers that generalize well to test data. An ensemble of deep ResNets achieved a 3.57% top-5 error on the ImageNet test set, setting a new state-of-the-art at the time.

3. Faster and more stable convergence compared to plain nets as network depth increased. Whereas plain net training error degraded beyond 20 layers, ResNets showed excellent convergence even with 100+ layers.

4. Identity shortcuts were shown to be sufficient; projection shortcuts did not improve accuracy. This indicates that identity mappings are crucial for propagating gradients, while projections are not essential for addressing the vanishing gradient problem.

Discussion

Impact

The ResNet architecture was hugely influential and is now considered a canonical neural network design. ResNets are frequently used as backbone feature extractors for a variety of computer vision tasks beyond image classification, including:

• Object detection (e.g. faster R-CNN, YOLO)
• Segmentation (e.g. Mask R-CNN)
• Action recognition
• Pose estimation

The residual learning concept has also been applied to other types of neural networks like recurrent neural networks for sequence modeling tasks.

Subsequent Developments

ResNet‘s success inspired many subsequent CNN designs that built upon the idea of skip connections:

• Wide Residual Networks (WRNs): Showed that increasing width is more effective than depth past 100+ layers. A 16-layer-deep WRN outperformed a 100-layer-deep thin ResNet.
• ResNeXt: Introduced a "cardinality" dimension to bottlneck blocks in addition to depth and width. Improves efficiency.
• DenseNet: Uses dense skip connections where each layer is connected to every other layer in a feed-forward fashion. Alleviates vanishing gradients and encourages feature reuse.
• Squeeze-and-Excitation Networks: Integrates a channel attention mechanism into ResNet blocks that adaptively recalibrates channel-wise feature maps. Boosts representational power.

Many of the top entries to the ImageNet competition in the years since 2015 have been variants of ResNet models.

Open Questions

While ResNet was certainly a breakthrough, it also raised new questions:

1. What are the optimal hyperparameters for ResNet training (depth, width, cardinality, etc.)? While many variants achieved excellent results, it‘s still not completely clear what an "ideal" network should look like.

2. How can residual learning be combined with other CNN enhancements like attention mechanisms, feature pyramids, or neural architecture search?

3. To what extent can extremely deep ResNets (>200 layers) offer further gains? There is evidence of diminishing returns and a maximum useful depth.

4. How well does the residual learning paradigm extend beyond computer vision to other domains like natural language processing or reinforcement learning?

It‘s likely that future research will continue to refine and enhance the residual learning paradigm introduced by ResNet.

Conclusion

Since its introduction in 2015, the Deep Residual Network (ResNet) architecture has become one of the most influential and widely used neural network designs in computer vision. By utilizing skip connections to propagate gradients directly across many layers, ResNets alleviate the vanishing gradient problem and unlock the potential of very deep networks.

The results speak for themselves – ResNets achieved state-of-the-art accuracy on challenging benchmarks like ImageNet while using fewer parameters and computations than competing architectures like VGGNet. Beyond simply enabling deeper networks, the residual learning paradigm seems to lead to faster, more stable training and better generalization.

In a sense, ResNet was the spark that lit the fuse for a flurry of innovation and progress in computer vision over the past several years. ResNet and its many variants are now ubiquitous as feature extractors for all sorts of vision tasks. At the same time, there is still much to be discovered about the full potential of residual learning and how to combine it with other key ideas in deep learning.

One thing is certain – ResNet has secured its place in the pantheon of groundbreaking neural network architectures alongside the likes of LeNet, AlexNet, and others. Its simple yet powerful core idea will continue to guide and inspire deep learning researchers and practitioners for years to come.