Understanding ResNet Architectures and Performance on CIFAR-10

Deep learning has revolutionized computer vision in the past decade, with convolutional neural networks (CNNs) achieving remarkable performance on challenging benchmarks like ImageNet. However, a number of obstacles have historically limited the depth to which CNNs can be scaled. In particular, deep networks become difficult to train due to the vanishing gradient problem, and counterintuitively, simply stacking more layers often leads to higher training error – a phenomenon known as the degradation problem.

To overcome these challenges and build ultra-deep networks, He et al. introduced deep residual learning in their seminal 2015 paper that proposed ResNet. The key innovation was residual blocks that learn residual functions with reference to the layer inputs, implemented via "skip" or "shortcut" connections. Stacking residual blocks enabled training stable networks of over 100 layers, far deeper than what was previously possible.

In this post, we‘ll dive into the ResNet architecture, focusing in particular on the compact ResNet-18 model that is widely used for smaller datasets like CIFAR-10. We‘ll examine ResNet-18‘s structure in detail and compare it to other ResNet variants and the related ResNeXt architecture. Finally, we‘ll analyze the performance of ResNet models of various depths trained on the CIFAR-10 dataset, demonstrating the accuracy gains made possible by extending ResNets to extreme depths.

The Degradation Problem

Let‘s first look more closely at the degradation problem that motivated ResNets. In theory, deeper networks should be strictly more powerful than shallower ones – in the worst case, extra layers could simply learn the identity mapping and match the shallower network‘s performance. However, in practice, adding more layers to plain convolutional networks leads to higher training error once networks become sufficiently deep:

Paradoxically, the 56-layer network has higher error on both the training and test sets compared to the 20-layer network, indicating an optimization difficulty rather than overfitting. The degradation problem was identified by He et al. as a major barrier to building very deep networks.

Residual Learning

ResNets solve the degradation problem through residual learning. In a residual block, layers learn a residual function F(x) with respect to the block‘s input x. The input x is added back to the output of the block‘s layers through a skip connection:

Denoting the underlying mapping the block fits as H(x), we have:

H(x) = F(x) + x

This can be viewed as the layers and skip connection together fitting the residual F(x) = H(x) – x, the difference between the true underlying mapping and the identity x. If the optimal function is closer to the identity, it will be easier to fit the residual. Empirically, residual functions are indeed easier to optimize than the unreferenced mappings.

Residual blocks can be stacked to arbitrary depths. When deeper models start converging, they are not worse than shallower counterparts, avoiding the degradation problem. As layer number increases, the training error of residual nets continues decreasing, in contrast to plain networks:

Additionally, skip connections provide a clear path for gradients to flow back to earlier layers, mitigating the vanishing gradients problem. Overall, residual learning enables training much deeper networks.

ResNet Architectures

ResNet refers to a family of architectures following the same basic template. All ResNets begin with a convolutional layer followed by a max pooling layer. The main body consists of a stack of residual blocks, divided into four stages with progressively more feature maps. Spatial downsampling is performed at the beginning of each stage by the first residual block, which has a convolutional layer with stride 2 on the skip connection. After the final stage, an average pooling layer and fully-connected layer produce the predictions.

The primary ResNet architectures are ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152, with the numbers denoting the layer count. ResNet-18 and ResNet-34 use the basic residual block with two 3×3 convolutional layers:

ResNet-50/101/152 use a "bottleneck" block design that is more computationally efficient for deeper models. It consists of 1×1, 3×3, and 1×1 convolutions, where the 1×1 layers reduce and restore dimensions to make the 3×3 layer‘s computation less expensive:

Focusing on ResNet-18

Let‘s now examine the exact layer specifications of ResNet-18, a compact model that is a strong baseline for smaller datasets like CIFAR-10. Here is the full ResNet-18 structure:

Some key observations:

• The first conv1 layer has a large 7×7 kernel and stride 2, quickly reducing spatial dimensions. Max pooling further downsamples to 1/4 the input resolution.
• There are 4 stages consisting of 2 basic residual blocks each. The first block of each stage has stride 2 to downsample.
• The channel dimensions are doubled at each stage, starting from 64 and increasing to 512. Spatial dimensions are halved.
• Average pooling after the final stage reduces to a 1×1 spatial size before the fully-connected layer.
• In total there are 18 learnable layers: 17 convolutional layers and 1 fully-connected layer.

When adapting ResNet-18 for CIFAR-10 rather than ImageNet, two modifications are typically made:

1. Remove the initial max pooling layer, since CIFAR-10 images are already small (32×32).
2. Replace the fully-connected layer with a 10-unit layer for the 10 CIFAR-10 classes.

Comparison to ResNeXt

An interesting follow-up to ResNet is ResNeXt, which utilizes grouped convolutions in the residual blocks. In a grouped convolution, input and output channels are divided into groups and convolutions are performed separately for each group. ResNeXt incorporates this idea via a "split-transform-merge" strategy inside bottleneck blocks:

The 3×3 layer is replaced by a grouped 3×3 convolution. This increases the model‘s cardinality (number of transformations) for a given parameter count. Interestingly, ResNeXt performs slightly better than ResNet when matched for number of parameters, suggesting the split-transform-merge paradigm and increased cardinality are beneficial.

Performance on CIFAR-10

To demonstrate the impact of residual learning and assess accuracy gains from increasing depth, let‘s compare the performance of various ResNet models on CIFAR-10. We‘ll evaluate ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152.

For training, we use stochastic gradient descent with momentum for 200 epochs. The initial learning rate is set to 0.1 and decreased by a factor of 10 after 1/2 and 3/4 of training. Data augmentation including random cropping and horizontal flipping is applied. Here are the training loss curves:

The deeper models have lower training loss and converge faster due to the residual learning framework. ResNet-18 ends up with the highest training loss, while ResNet-152 achieves the lowest.

Next let‘s look at test accuracy:

There is a clear trend of accuracy increasing as the ResNet depth is extended. ResNet-18 reaches 94.6% accuracy, while ResNet-152 achieves 95.7% – over a 1% improvement solely from adding more depth. This demonstrates the effectiveness of residual learning in enabling ultra-deep models that continue to enhance performance.

Conclusion

ResNet was a transformative innovation in deep learning, overcoming major barriers to training extremely deep networks. Skip connections in residual blocks solved the degradation problem, with deeper ResNets achieving lower training error than shallower counterparts rather than higher error. Residual learning also helps gradients flow to bottom layers, mitigating vanishing gradients.

On CIFAR-10, we observe accuracy consistently increases with depth from ResNet-18 to ResNet-152. The deepest 152-layer network attains over 1% higher accuracy than the shallowest 18-layer model. However, the accuracy gains are relatively marginal considering the substantial increase in parameters and computational cost.

In summary, ResNet demonstrates the power of deep residual learning to build highly expressive models. The general concept has been extended in various ways, such as increased cardinality in ResNeXt. ResNet architectures remain widely used today and have had a profound impact on computer vision. Residual connections are a key component in more recent state-of-the-art backbones like the Vision Transformer.