Building ResNet from Scratch with Python: A Comprehensive Guide
Introduction
Deep learning has revolutionized computer vision and achieved remarkable breakthroughs in tasks like image classification, object detection, and segmentation. One of the most influential architectures that has played a pivotal role in this progress is the Residual Network, or ResNet for short.
ResNet, introduced by Kaiming He et al. in their 2015 paper "Deep Residual Learning for Image Recognition," has become a go-to choice for many vision tasks due to its ability to train very deep networks effectively. By the end of this article, you‘ll have a solid understanding of ResNet and be able to implement it from scratch using Python. Let‘s dive in!
The Need for ResNet
As deep learning models grew deeper with the aim of learning more complex features, a peculiar problem emerged—the accuracy saturated and then rapidly degraded as depth increased. This wasn‘t caused by overfitting, but by the difficulty in optimizing deep networks due to vanishing/exploding gradients.
ResNet addressed this issue by introducing a simple yet effective solution: residual blocks with skip connections. These connections allow the gradients to flow directly through the network, mitigating the vanishing gradient problem. As a result, ResNets can be hundreds or even thousands of layers deep and still achieve impressive performance.
ResNet Architecture
At the heart of ResNet are the residual blocks. A residual block consists of a few convolutional layers, followed by a skip connection that adds the input to the output of the block. Mathematically, if the input to a residual block is x, and the learned function is F(x), then the output of the block is:
output = F(x) + x
This identity mapping allows the network to learn residual functions with reference to the input, which is easier than learning the original, unreferenced functions.
ResNet comes in different flavors based on the number of layers:
- ResNet-18 and ResNet-34 use basic residual blocks with two convolutional layers.
- ResNet-50/101/152 use bottleneck blocks with three convolutional layers, which are more computationally efficient.
The overall ResNet architecture starts with an initial convolutional layer, followed by a stack of residual blocks organized into several stages. The number of blocks per stage increases as we go deeper into the network. Finally, an average pooling layer and a fully connected layer are used for classification.
Implementing ResNet with Python
Now let‘s see how to build ResNet from scratch using Python. We‘ll use PyTorch, but the concepts are applicable to any deep learning framework.
First, we define the basic building block—a residual block with two convolutional layers:
class BasicBlock(nn.Module):
def __init__(self, in_channels, out_channels, stride=1):
super().__init__()
self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3,
stride=stride, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(out_channels)
self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3,
stride=1, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(out_channels)
# Skip connection
if stride != 1 or in_channels != out_channels:
self.shortcut = nn.Sequential(
nn.Conv2d(in_channels, out_channels, kernel_size=1,
stride=stride, bias=False),
nn.BatchNorm2d(out_channels)
)
else:
self.shortcut = nn.Identity()
def forward(self, x):
out = nn.ReLU()(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
out += self.shortcut(x)
out = nn.ReLU()(out)
return outThe BasicBlock consists of two convolutional layers with batch normalization and ReLU activation. The skip connection is implemented using either a convolutional layer (if input/output dimensions differ) or an identity mapping.
Next, we define the ResNet class that puts together the residual blocks to form the complete network:
class ResNet(nn.Module):
def __init__(self, block, layers, num_classes=1000):
super().__init__()
self.in_channels = 64
self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)
self.bn1 = nn.BatchNorm2d(64)
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
self.layer1 = self._make_layer(block, 64, layers[0])
self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
self.fc = nn.Linear(512, num_classes)
def _make_layer(self, block, out_channels, blocks, stride=1):
layers = []
layers.append(block(self.in_channels, out_channels, stride))
self.in_channels = out_channels
for _ in range(1, blocks):
layers.append(block(out_channels, out_channels))
return nn.Sequential(*layers)
def forward(self, x):
x = self.maxpool(nn.ReLU()(self.bn1(self.conv1(x))))
x = self.layer1(x)
x = self.layer2(x)
x = self.layer3(x)
x = self.layer4(x)
x = self.avgpool(x)
x = torch.flatten(x, 1)
x = self.fc(x)
return xThe ResNet class takes the block type (BasicBlock in this case), the number of blocks per stage, and the number of classes as input. It defines the initial convolutional layer, followed by four stages of residual blocks. The _make_layer function is a helper to create each stage by stacking multiple residual blocks. Finally, an adaptive average pooling layer and a fully connected layer are used for classification.
We can instantiate different ResNet configurations by specifying the block type and the number of blocks per stage:
def resnet18():
return ResNet(BasicBlock, [2, 2, 2, 2])
def resnet34():
return ResNet(BasicBlock, [3, 4, 6, 3])Training and Evaluation
To train ResNet, we follow the standard training loop in PyTorch. We first prepare the data using DataLoader, define the loss function (e.g., cross-entropy) and optimizer (e.g., SGD), and then iterate over the dataset for a specified number of epochs.
Here‘s a basic training loop:
def train(model, dataloader, criterion, optimizer, device):
model.train()
running_loss = 0.0
for images, labels in dataloader:
images, labels = images.to(device), labels.to(device)
optimizer.zero_grad()
outputs = model(images)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
running_loss += loss.item() * images.size(0)
epoch_loss = running_loss / len(dataloader.dataset)
return epoch_lossTo evaluate the trained model, we use a similar loop but without the gradient computation and optimization steps:
def evaluate(model, dataloader, criterion, device):
model.eval()
running_loss = 0.0
correct = 0
with torch.no_grad():
for images, labels in dataloader:
images, labels = images.to(device), labels.to(device)
outputs = model(images)
loss = criterion(outputs, labels)
running_loss += loss.item() * images.size(0)
_, predicted = torch.max(outputs, 1)
correct += (predicted == labels).sum().item()
epoch_loss = running_loss / len(dataloader.dataset)
epoch_accuracy = correct / len(dataloader.dataset)
return epoch_loss, epoch_accuracyIt‘s important to use techniques like learning rate scheduling, weight decay, and data augmentation to train ResNet effectively. Experimenting with different optimizers, initializations, and regularization methods can also help improve performance.
Applications and Extensions
ResNet has been widely adopted for various vision tasks beyond image classification. It serves as a powerful backbone for object detection (e.g., Faster R-CNN), semantic segmentation (e.g., DeepLab), and pose estimation. Transfer learning with pre-trained ResNet weights is also common, as it allows leveraging the learned features for tasks with limited training data.
Since the introduction of ResNet, many variants and improvements have been proposed. Some notable ones include:
- Wide ResNet: Increases the width (number of channels) of residual blocks for improved performance.
- ResNeXt: Introduces grouped convolutions and aggregated transformations for better parameter efficiency.
- DenseNet: Connects each layer to every other layer in a feed-forward fashion, strengthening feature propagation.
- SENet: Incorporates squeeze-and-excitation blocks to adaptively recalibrate channel-wise feature responses.
These extensions build upon the core ideas of ResNet and push the boundaries of deep learning even further.
Conclusion
ResNet has undoubtedly left a significant mark on the field of deep learning. By alleviating the vanishing gradient problem through residual connections, it has enabled the training of extremely deep networks with remarkable performance.
In this article, we explored the motivation behind ResNet, dissected its architecture, and learned how to implement it from scratch using Python and PyTorch. We also briefly touched upon its applications and recent extensions.
As you venture into your own deep learning projects, consider leveraging the power of ResNet. Experiment with different configurations, apply transfer learning, and don‘t hesitate to dive into the latest research to stay up-to-date with the ever-evolving landscape of deep learning.
Remember, the key to mastering deep learning is practice and perseverance. Keep coding, keep exploring, and most importantly, enjoy the journey!
![A Comprehensive Guide to Handling Missing Values in Data Analysis [2026 Update]](https://33rdsquare.com/wp-content/uploads/2024/12/20241213112737-675c1a29eb3e6-768x380.jpg)
