Building ResNets from Scratch in TensorFlow

Deep learning has revolutionized the field of computer vision, enabling machines to recognize images and objects with remarkable accuracy. One of the key innovations behind this progress is the development of very deep convolutional neural networks (CNNs). In particular, Residual Networks (ResNets) allow CNNs to be trained to extreme depths by introducing skip connections and bottleneck architectures. In this post, we‘ll dive into the details of ResNets and implement one from scratch in TensorFlow.

Why ResNets?

The core idea behind CNNs is to stack many convolutional and pooling layers to allow the network to learn increasingly complex and abstract visual features. For a long time, the rule of thumb was that deeper networks would achieve better accuracy on challenging benchmarks like ImageNet.

However, researchers observed a counterintuitive phenomenon – as networks grew beyond a certain depth (usually around 20-30 layers), training became increasingly difficult and test error actually began to degrade. This wasn‘t due to overfitting, as the training error also increased. The problem was the difficulty of optimizing such deep networks due to vanishing gradients and the fragility of deep linear chains of layers.

ResNets, introduced by He et al. in a 2015 paper, offered an ingenious solution. The key was to add shortcut or skip connections that allow the signal to bypass one or more layers. Formally, if the desired mapping of a stack of layers is H(x), the network instead learns the residual mapping F(x) = H(x) – x, which is easier to optimize. The original input x is then added back to the output F(x) to recover the desired mapping: H(x) = F(x) + x.

ResNets also introduced bottleneck architectures for the residual blocks that use 1×1 convolutions to first reduce and then restore dimensions. This improves computational efficiency.

With these innovations, ResNets with 50, 101, 152 and even 1000+ layers were successfully trained, setting new records on ImageNet and other benchmarks. ResNets are now widely used as a backbone for many computer vision tasks like object detection and segmentation.

Implementing ResNets in TensorFlow

Let‘s see how to code up a ResNet in TensorFlow using the Keras functional API. We‘ll build things step-by-step, starting with the identity and bottleneck blocks.

Identity Block

The identity block has the property that the activation dimensions are the same for the input and output. This allows the input to be directly added to the output to form the residual connection.

Here‘s the code for an identity block that performs two 3×3 convolutions with a specified number of filters, with batch normalization and ReLU activations applied after each convolution:

def identity_block(x, filters):

    x_skip = x

    # Layer 1
    x = Conv2D(filters, (3,3), padding = ‘same‘)(x)
    x = BatchNormalization(axis=3)(x)
    x = Activation(‘relu‘)(x)

    # Layer 2
    x = Conv2D(filters, (3,3), padding = ‘same‘)(x)
    x = BatchNormalization(axis=3)(x)

    # Add Residue
    x = Add()([x, x_skip])     
    x = Activation(‘relu‘)(x)

    return x

Convolutional Block

The convolutional block is used when the input and output dimensions don‘t match up. The difference with the identity block is that the input is passed through a 1×1 convolution with stride 2 to reduce the dimensions before being added to the output.

def convolutional_block(x, filters):

    x_skip = x

    # Layer 1
    x = Conv2D(filters, (3,3), padding = ‘same‘, strides = (2,2))(x)
    x = BatchNormalization(axis=3)(x)
    x = Activation(‘relu‘)(x)

    # Layer 2
    x = Conv2D(filters, (3,3), padding = ‘same‘)(x)
    x = BatchNormalization(axis=3)(x)

    # Processing Residue with conv(1,1)
    x_skip = Conv2D(filters, (1,1), strides = (2,2))(x_skip)

    # Add Residue
    x = Add()([x, x_skip])     
    x = Activation(‘relu‘)(x)

    return x

Building the ResNet Model

With the identity and convolutional blocks in place, we can now build a full ResNet model. The architecture follows this pattern:

1. Initial convolutional layer with 64 filters and 7×7 kernel
2. Max pooling layer with 3×3 pool size and stride 2
3. Stack of residual blocks with increasing filter sizes:
   • 3 identity blocks with 64 filters
   • 1 convolutional block and 3 identity blocks with 128 filters
   • 1 convolutional block and 5 identity blocks with 256 filters
   • 1 convolutional block and 2 identity blocks with 512 filters
4. Average pooling layer with pool size equal to the input dimensions
5. Flatten the output and add a fully-connected layer for classification

Here‘s how that looks in code for a 50-layer ResNet:

def ResNet50(input_shape = (224, 224, 3), classes = 1000):

    # Stage 1
    x_input = Input(input_shape)
    x = ZeroPadding2D((3, 3))(x_input)
    x = Conv2D(64, (7, 7), strides = (2, 2))(x)
    x = BatchNormalization()(x)
    x = Activation(‘relu‘)(x)
    x = MaxPool2D((3, 3), strides=(2, 2))(x)

    # Stage 2
    x = convolutional_block(x, filters = 64)
    x = identity_block(x, filters = 64)
    x = identity_block(x, filters = 64)

    # Stage 3
    x = convolutional_block(x, filters = 128)
    x = identity_block(x, filters = 128)
    x = identity_block(x, filters = 128)
    x = identity_block(x, filters = 128)

    # Stage 4
    x = convolutional_block(x, filters = 256)
    x = identity_block(x, filters = 256)
    x = identity_block(x, filters = 256)
    x = identity_block(x, filters = 256)
    x = identity_block(x, filters = 256)
    x = identity_block(x, filters = 256)

    # Stage 5
    x = convolutional_block(x, filters = 512)
    x = identity_block(x, filters = 512)
    x = identity_block(x, filters = 512)

    # Final layers
    x = GlobalAveragePooling2D()(x)
    x = Dense(classes, activation=‘softmax‘)(x)

    model = Model(inputs = x_input, outputs = x, name = "ResNet50")

    return model

You can easily modify this code to build ResNets with different depths like 101 or 152 layers by adding more identity blocks in each stage.

The model can be compiled and trained as usual in Keras, for example:

model = ResNet50(input_shape = (32, 32, 3), classes = 10)

model.compile(optimizer=‘adam‘, loss=‘categorical_crossentropy‘, metrics=[‘accuracy‘])

model.fit(x_train, y_train, batch_size=32, epochs=100, verbose=1, validation_data=(x_test, y_test))

Tips for Training ResNets

While ResNets make it possible to effectively train extremely deep networks, there are still some challenges and best practices to keep in mind:

1. Use an appropriate learning rate schedule. The original ResNet paper used a learning rate starting at 0.1 and divided by 10 when the validation error plateaued. Newer schedules like cosine annealing can improve convergence.

2. Data augmentation is crucial for achieving top results, especially on smaller datasets. Techniques like random cropping, flipping, and scaling can greatly improve generalization.

3. Label smoothing is another regularization technique that can boost accuracy. Instead of using hard 0 and 1 targets, a small constant like 0.1 is subtracted from 1 for the true class and distributed to the false classes.

4. Be mindful of batch sizes and memory requirements, especially when training on GPUs. ResNets, particularly with bottlenecks, are more memory-efficient than plain stacks of convolutions. But the activations from deep networks can still consume a lot of memory.

5. Consider pre-training on a large dataset like ImageNet if your target dataset is small. The weights of a ResNet trained on ImageNet serve as a great initialization that can significantly speed up convergence.

Comparing ResNets to Other Architectures

ResNets were a pioneering innovation in deep learning for computer vision, but they‘re certainly not the only effective CNN architecture. Other notable models include:

• VGG: A simple but effective stack of 3×3 convolutions and max pooling developed by Oxford‘s Visual Geometry Group. Widely used as a feature extractor.

• Inception: Introduced by Google, Inception nets use parallel branches with different convolution and pooling operations to capture information at various scales.

• DenseNet: Takes the idea of skip connections to the extreme by connecting each layer to every other layer in a dense block. Improves feature reuse and reduces the number of parameters.

• EfficientNet: A recent family of models developed by Google that systematically scale network width, depth, and resolution to achieve state-of-the-art efficiency.

ResNets remain a popular backbone architecture for many applications due to their simplicity and proven performance. The introduction of skip connections was a key step in the quest to scale neural networks to greater depths and capabilities.

Conclusion

ResNets are a powerful and widely-used CNN architecture that enables the training of extremely deep networks. By introducing identity skip connections and bottleneck designs, ResNets overcome the problems of vanishing gradients and allow the effective learning of very complex visual features.

In this post, we saw how to implement the building blocks of ResNets in TensorFlow and Keras, and how to put them together into a full 50-layer ResNet model. We also discussed some tips and tricks for achieving top performance with ResNets.

The core ideas behind ResNets have been instrumental in advancing the state-of-the-art in computer vision. Skip connections have been widely adopted in other models, and the ability to successfully scale networks to 100+ layers has opened up new frontiers in visual understanding.

I encourage you to experiment with building ResNets and other cutting-edge CNN architectures. Feel free to use and adapt the code presented here. With the power of deep learning frameworks like TensorFlow, it‘s easier than ever to construct these sophisticated models and train them on challenging datasets.

I hope this post has demystified some of the magic behind ResNets and inspired you to dive deeper into deep learning for computer vision. Happy coding!