Efficient and Accurate Object Detection with Lightweight Decoder-Free Transformers

ByJordan Brown

Introduction

Object detection, the task of identifying and localizing objects of interest in an image, is a fundamental problem in computer vision with wide-ranging applications from autonomous driving to video surveillance. The main challenge lies in effectively representing objects at vastly different scales – from small objects like a cellphone to large objects like a bus.

Traditional object detectors rely on multi-scale feature pyramids or two-stage region proposal networks to handle scale variation, but these hand-engineered pipelines are complex and computationally expensive. Recently, transformer-based architectures like DETR have emerged as a promising alternative, using a set-based global reasoning framework to directly predict object bounding boxes in a single forward pass. However, DETR still requires a decoder with cross-attention that demands significant training time to converge.

To address these limitations, researchers have proposed a novel Decoder-Free Fully Transformer-based (DFFT) object detector that achieves high efficiency during both training and inference. DFFT introduces a lightweight single-stage detection pipeline centered around two key innovations:

  1. Replacing the costly decoder with two strong encoder branches to maintain the accuracy of single-scale feature map prediction
  2. Leveraging low-level features with rich semantics for improved small object detection

In this post, we‘ll dive into the technical details of DFFT and explore how this decoder-free transformer architecture pushes the state-of-the-art in efficient object detection.

Decoder-Free Fully Transformer Detection

Overview

The DFFT object detector consists of three main components, as illustrated in Figure 1:

  1. Detection-Oriented Transformer (DOT) backbone
  2. Scale-Aggregated Encoder (SAE)
  3. Task-Aligned Encoder (TAE)

The DOT backbone is a lightweight feature extractor that outputs multi-scale feature maps with strong semantics. The SAE then progressively fuses these features into a single high-resolution representation through cross-scale attention. Finally, the TAE handles conflicts between classification and regression tasks in the dense prediction head to produce the final object detections.

By eliminating the decoder and unifying the multi-scale features, DFFT greatly simplifies the object detection pipeline into an encoder-only architecture optimized for both accuracy and speed. Next we‘ll look at each component in more detail.

Detection-Oriented Transformer Backbone

The DOT backbone is designed to extract semantically rich multi-scale features suitable for object detection. As shown in Figure 2, it consists of a patch embedding layer followed by four stacked DOT stages. Each stage contains a Semantic-Augmented Attention (SAA) module that strengthens the low-level features by aggregating semantic information from the deeper layers.

Specifically, the SAA module performs cross-attention between the low-level features and high-level features within a local window. This allows the low-level features to incorporate more contextual information useful for localizing small objects, without significantly increasing computation.

After the last DOT stage, we obtain a pyramid of feature maps at four scales: 1/4, 1/8, 1/16, and 1/32 of the input image resolution. These multi-scale features are then input to the SAE for further processing.

Scale-Aggregated Encoder

The SAE is responsible for aggregating the multi-scale DOT features into a single high-resolution representation. It consists of three SAE blocks operating sequentially at the 1/16, 1/8, and 1/4 scales. As illustrated in Figure 3, each block performs multi-head self-attention to capture long-range dependencies, followed by a convolutional feed-forward network.

To fuse features across scales, the SAE block first applies a 1×1 convolution to the lower-resolution features, then upsamples them to the higher target resolution through bilinear interpolation. These upsampled features are concatenated with the target-resolution features and passed through another 1×1 convolution to align the channels. This cross-scale feature aggregation allows the SAE to incorporate both high-level semantics and low-level details into the final representation.

After the last SAE block, we obtain a single feature map at 1/4 resolution, which contains rich information for detecting objects of all sizes. This unified representation greatly simplifies the subsequent detection head, avoiding the need to make independent predictions from each scale.

Task-Aligned Encoder

The TAE aims to resolve the conflict between classification and regression tasks in the final dense object detection head. Previous studies have shown that simply sharing features between the two tasks leads to suboptimal performance, as classification benefits more from high-level semantic features while regression requires low-level spatial details.

To address this issue, the TAE first splits the aggregated SAE features into two streams using group convolution, as shown in Figure 4. The classification stream undergoes a series of global self-attention and convolutional blocks to extract class-specific features. In parallel, the box regression stream performs local self-attention to preserve precise spatial information.

At the end of the TAE, the two streams are concatenated and fused using a 1×1 convolution, producing a unified feature map optimized for both tasks. This task-aligned representation enables the final detection head to make joint predictions for classification and regression, improving both accuracy and inference speed compared to decoupled designs.

Experiments

The authors evaluate DFFT on the popular MS COCO benchmark for object detection. They compare two models, DFFT and DFFT-Light, against a variety of CNN and transformer baselines, focusing on the trade-off between accuracy and efficiency.

Table 1 shows the main results on COCO test-dev. DFFT achieves an impressive 49.2 AP while being 60% faster than the ResNet-101 Faster R-CNN baseline. DFFT-Light offers an even better balance, matching the ResNet-50 Faster R-CNN accuracy with 7x fewer parameters and 6x less computation. These results demonstrate the strong performance of the decoder-free transformer architecture for object detection.

The authors also provide an in-depth ablation study to validate the impact of each proposed component. Key findings include:

  • The DOT backbone effectively captures multi-scale features and outperforms the ResNet backbone by 2-4 AP across model sizes.

  • The SAE provides significant benefits by aggregating features across scales, improving over the baseline by 1.5-2 AP. Increasing the SAE blocks further boosts accuracy.

  • The TAE brings 1-1.5 AP gain over shared classification and regression features by aligning the two tasks. Using group convolution is more effective than a simple split.

  • Reducing the number of detection heads leads to a good balance between accuracy and speed. The best configuration uses one classification head and three regression heads.

Conclusion

This work presents DFFT, a novel decoder-free transformer architecture for efficient and accurate object detection. By eliminating the costly decoder and exploiting the expressivity of transformers, DFFT unifies the multi-scale features into a single high-resolution representation that is simultaneously optimized for classification and box regression. Extensive experiments show that DFFT pushes the boundary of accuracy-efficiency trade-off, outperforming competitive CNN and transformer detectors across model sizes.

The core ideas of DFFT are general and can be applied to many other vision tasks that require dense prediction, such as instance segmentation, keypoint detection, and panoptic segmentation. In addition, the design of the DOT backbone and the SAE could potentially benefit other visual recognition problems like semantic segmentation and image classification.

There are multiple promising directions for future research on decoder-free transformers for object detection. One is to further scale up the training data and model size to improve performance on large-vocabulary detection tasks. Another direction is to investigate better feature fusion and alignment strategies for jointly optimizing classification and regression in a one-stage framework. Finally, integrating DFFT with downstream tasks like object tracking and action recognition could lead to more powerful and efficient perception systems.

Object detection has been a central problem in computer vision for decades, with active research continuing to push the state of the art. The transformer architecture has brought a new paradigm shift in the field, offering a unified and flexible framework for capturing long-range dependencies and achieving global reasoning. DFFT demonstrates the great potential of decoder-free transformers for efficient and accurate object detection, paving the way for future research on this exciting direction.

References

[1] Yao, Lewei, et al. "DFFT: Decoder-Free Fully Transformer Network for Object Detection." arXiv preprint arXiv:2110.08049 (2021).

[2] Carion, Nicolas, et al. "End-to-End Object Detection with Transformers." ECCV 2020.

[3] Zhu, Xizhou, et al. "Deformable DETR: Deformable Transformers for End-to-End Object Detection." ICLR 2021.

[4] Dosovitskiy, Alexey, et al. "An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale." ICLR 2021.

Code and Resources

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

You May Like to Read,

Similar Posts