RetinaNet: Advancing the State of the Art in Object Detection
Object detection is a central problem in computer vision, with far-reaching applications in areas like autonomous driving, surveillance, and robotics. The goal is to accurately localize and classify objects of interest within an image. In recent years, deep learning has revolutionized object detection, with a variety of novel architectures pushing the boundaries of speed and accuracy.
One of the most impactful developments has been RetinaNet, a single-stage dense object detector proposed by researchers at Facebook AI Research (FAIR) in 2017. RetinaNet achieved state-of-the-art results on the challenging COCO benchmark, surpassing more complex two-stage detectors like Faster R-CNN while being simpler and faster. In this blog post, we‘ll dive deep into the workings of RetinaNet and understand the key innovations that enable its impressive performance. We‘ll also discuss recent advancements and extensions to the architecture.
The Anatomy of RetinaNet
At a high level, RetinaNet is composed of three main components:
- A backbone convolutional neural network for computing feature maps at multiple scales
- A classification subnet for predicting the probability of object presence at each spatial position for each anchor and object class
- A regression subnet for regressing the offset from each anchor box to a nearby ground-truth object, if one exists
The backbone network is responsible for generating a multi-scale convolutional feature pyramid over a single input image resolution. RetinaNet uses a Feature Pyramid Network (FPN) architecture for its backbone, which was introduced in a previous paper by the same authors [1]. The key idea of FPN is to leverage the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost.
Specifically, FPN combines low-resolution, semantically strong features with high-resolution, semantically weak features via a top-down pathway and lateral connections:
- A bottom-up pathway that computes a feature hierarchy consisting of feature maps at several scales with a scaling step of 2
- A top-down pathway that hallucinates higher resolution features by upsampling spatially coarser, but semantically stronger, feature maps from higher pyramid levels
- Lateral connections between the bottom-up pathway and the top-down pathway that help the network learn richer, multi-scale feature representations
Attached to the backbone are two fully convolutional subnets – one for classifying anchor boxes and one for regressing from anchor boxes to ground-truth object boxes. The classification subnet predicts the probability of object presence at each spatial position for each of the A anchors and K object classes. It does this by applying four 3×3 conv layers, each with C filters and each followed by ReLU activations, followed by a final sigmoid layer.
The box regression subnet has an identical structure to the classification subnet, but instead predicts the 4 coordinates (x, y, width, height) encoding the location of the object relative to the anchor box. If an object exists at a given anchor, the subnet predicts an offset that refines the anchor box closer to the ground-truth box. The regression subnet uses a robust smooth L1 loss function.
Focal Loss: Addressing Foreground-Background Imbalance
One of the key insights of RetinaNet is addressing the extreme foreground-background class imbalance encountered during training of dense detectors. The authors observe that the vast majority of anchors typically contain no objects and contribute no useful learning signal. This imbalance causes two problems:
- Training is inefficient as most negative anchors are easy to classify and don‘t contribute to convergence
- The easy negatives can overwhelm the loss and cause model instability
To remedy this, RetinaNet employs a novel loss function called focal loss. Focal loss is a modified form of cross entropy loss that down-weights the contribution of easy examples, enabling the model to focus learning on hard negatives. The focal loss (FL) function is defined as:
$FL(p_t) = -(1 – p_t)^\gamma log(p_t)$
where $p_t$ is the model‘s estimated probability for the ground-truth class and $\gamma$ is a tunable focusing parameter.
When an example is misclassified and $p_t$ is small, the modulating factor $(1 – p_t)^\gamma$ is near 1 and the loss is unaffected. However, as $p_t \rightarrow 1$ (i.e. the model is very confident in the correct class), the factor approaches 0 and down-weights the contribution of well-classified examples. Typically $\gamma=2$ works well in practice.
Focal loss has a dramatic impact on training RetinaNet. The figures below plot the empirical cumulative distribution functions of the normalized loss for positive and negative anchors with cross entropy (CE) loss and focal loss (FL). With CE loss, the vast majority of anchors have a very low loss and consequently make a small contribution to the total loss. In contrast, with focal loss the distribution is heavily concentrated to the right, with most examples having non-trivial loss values.
Intuitively, focal loss focuses training on a sparse set of hard examples and prevents the overwhelming number of easy negatives from dominating the loss. This enables the model to more quickly and effectively learn to separate foreground from background.
Benchmarking RetinaNet
So how well does RetinaNet actually perform? The authors evaluate their model on the widely used COCO object detection dataset, which contains ~135k images and 80 object categories. They compare RetinaNet against several other state-of-the-art detectors, including two-stage models like Faster R-CNN with FPN and one-stage models like SSD and YOLO.
The results are impressive – using a ResNet-101-FPN backbone, RetinaNet achieves an mAP of 39.1 on the COCO test-dev set. This surpasses the performance of all previous single-stage detectors by a significant margin. RetinaNet also outperforms leading two-stage detectors like Faster R-CNN+++ [2], while being considerably simpler and faster.
In terms of computational efficiency, RetinaNet compares favorably to other detectors. The ResNet-50-FPN variant requires 3.8 GFLOPs and runs at 122 ms per image on a NVIDIA M40 GPU. This is roughly 3.5x faster than Faster R-CNN+++ [2] and more than 8x faster than Mask R-CNN [3]. The larger ResNet-101-FPN model achieves higher accuracy while still being faster than competing methods.
| Model | Backbone | mAP | GFLOPs | Time (ms) |
|---|---|---|---|---|
| Two-stage: | ||||
| Faster R-CNN+++ [2] | ResNet-101-C4 | 34.9 | 47.8 | 420 |
| Faster R-CNN w FPN [1] | ResNet-101-FPN | 36.2 | 73.0 | 172 |
| Mask R-CNN [3] | ResNeXt-101-FPN | 37.1 | – | 210 |
| One-stage: | ||||
| YOLO9000 [4] | Darknet-19 | 21.6 | 63.5 | 51 |
| SSD513 [5] | ResNet-101-SSD | 31.2 | – | 125 |
| DSSD513 [6] | ResNet-101-DSSD | 33.2 | – | 182 |
| RetinaNet (ours) | ResNet-50-FPN | 35.7 | 3.8 | 73 |
| RetinaNet (ours) | ResNet-101-FPN | 37.8 | 7.1 | 104 |
| RetinaNet (ours) | ResNeXt-32x8d-101-FPN | 40.1 | 6.0 | 198 |
Qualitatively, RetinaNet produces very clean and accurate bounding boxes, even for challenging images. Some example detections are shown below:
Recent Advancements
In the years since RetinaNet was introduced, there have been several notable developments that extend and improve upon the original model:
-
Deformable RetinaNet [7]: Enhances the backbone and detection subnets with deformable convolutions, which can adapt the receptive field to the size and shape of objects. Increases mAP to 46.4 on COCO.
-
NAS-FPN [8]: Employs neural architecture search to learn an optimal feature pyramid topology that makes more efficient use of multi-scale features. Achieves 48.3 mAP on COCO.
-
DetectoRS [9]: Combines a recursive feature pyramid, switchable atrous convolution, and a novel end-to-end bounding box refinement module to boost performance. Obtains a COCO mAP of 53.3.
-
EfficientDet [10]: Incorporates EfficientNet backbones, a bi-directional feature pyramid network, and a compound scaling method to create a family of object detectors that achieves state-of-the-art efficiency across a wide spectrum of resource constraints. Reaches 52.2 mAP on COCO with 4x-9x fewer parameters and 13x-42x less computation compared to previous detectors.
The rapid progress in object detection is a testament to the value of insightful design improvements like those introduced in RetinaNet. By identifying and targeting key issues like class imbalance, RetinaNet opened the door for simple, effective single-stage detection architectures.
Conclusion
RetinaNet is an elegant and impactful approach to object detection that combines a powerful FPN backbone with a novel focal loss function to achieve state-of-the-art accuracy. By addressing key challenges like foreground-background class imbalance, RetinaNet obtains performance on par with more complex two-stage detectors while offering a simpler, faster architecture.
RetinaNet has had a significant influence on the trajectory of modern object detection research. It has inspired many subsequent works that leverage its insights to push the boundaries of detection efficiency and accuracy. From a practitioner‘s perspective, RetinaNet is an excellent choice for a wide range of applications due to its simplicity, flexibility, and strong empirical performance.
As we continue to seek ever more powerful object detection models to enable next-generation perception systems, the core ideas behind RetinaNet – thoughtful architecture design, principled optimization objectives, and careful consideration of practical tradeoffs – will undoubtedly be key parts of the solution. Here‘s to the exciting road ahead!
