NanoDet Plus: Lightweight, Blazing-Fast Object Detection for the Edge

Object detection, the task of locating and classifying objects in images or video, is a fundamental problem in computer vision with wide-ranging applications. Self-driving cars need to spot pedestrians, animals, and other vehicles in real-time. Robots use object detection to navigate and interact with the world around them. Smartphones can search for products, translate text, or apply fun filters based on detecting relevant objects.

Traditionally, the most accurate object detectors have been large, compute-intensive models that run on powerful GPUs in the cloud or on high-end desktop hardware. But there is an increasing need for models that can run efficiently on mobile devices and edge computing platforms with limited memory and processing power. NanoDet is a family of cutting-edge object detection models optimized for these constrained environments.

Why Efficient Detection Matters

The world is becoming more mobile and connected every day. There are now over 6 billion smartphones in use globally, each one a potential platform for AI-enhanced applications. The edge AI market is projected to grow from $590 million in 2020 to $1.83 billion by 2026, a 20.7% CAGR (Mordor Intelligence). This growth is driven by demand for real-time, low-latency processing of sensor data closer to where it is generated, without the cost and privacy risks of sending everything to the cloud.

Object detection is a key enabling technology for many edge AI use cases:

• Smart cameras for security, retail analytics, factory inspection, etc. can detect suspicious activities, count customers, or spot product defects on the camera itself, reducing bandwidth usage and server costs.

• Visual search allows users to discover products, places, and information by snapping photos with their phone. Identifying relevant objects on-device makes the experience faster and more seamless.

• Augmented reality apps need to detect and track objects in the user‘s environment to overlay digital content and enable interactions. Offloading detection to a server would introduce unacceptable latency.

• Autonomous robots rely on object detection to make sense of their surroundings and plan actions. Processing video feeds locally avoids the saftey and reliability issues of remote control.

However, running state-of-the-art object detectors like Mask R-CNN or YOLOv4 on mobile devices is challenging due to their large size and high computational demands. These models can have over 100M parameters and require 10s of billions of FLOPs (floating-point operations) per inference. In contrast, a typical smartphone GPU may have only 1-2 GB of memory and be capable of 100-500 GFLOPS. Fitting the models on-device while achieving real-time speeds is difficult.

Inside NanoDet

NanoDet is designed from the ground up to address the challenges of efficient mobile detection. It builds on the popular FCOS framework, which divides the image into a grid and performs object classification and bounding box regression at each grid point. However, NanoDet incorporates several key innovations to significantly improve speed and accuracy over vanilla FCOS:

1. Ghost-PAN: The Feature Pyramid Network (FPN) is a common component of detectors that combines high-resolution features from lower layers of the backbone with semantically rich features from higher layers. However, standard FPNs can be computationally expensive for lightweight backbones. NanoDet replaces the FPN with Ghost-PAN, a novel lightweight feature pyramid that uses ghosted pointwise convolutions to reduce computation while maintaining accuracy. Ghost-PAN reduces the parameter count of the neck by 90% compared to FPN with negligible accuracy loss.

2. Assign Guidance Module (AGM): FCOS assigns positive/negative labels to grid points based on their distance to ground truth objects. However, the optimal assignment strategy may vary for different images and objects. NanoDet introduces AGM, a small network that adaptively learns the best assignment for each training example. AGM improves NanoDet‘s mAP by 1.1% on the COCO dataset.

3. Dynamic Soft Label Assigner (DSLA): In addition to hard 0/1 labels, NanoDet uses soft labels that capture the uncertainty of assignment decisions. DSLA dynamically generates these labels based on the relative distances between grid points and object centers. Soft labels provide richer training signals and are especially beneficial when distilling knowledge from a larger teacher model. DSLA boosts mAP by 0.5% over hard labels.

4. Generalized Focal Loss (GFL): NanoDet adopts the generalized focal loss from the GFL detector for both classification and bounding box regression. GFL reshapes the standard loss to better handle class imbalance and stabilize training. It outperforms the original FCOS loss functions across a range of backbones and model scales.

5. Quantization-friendly design: NanoDet is carefully designed to be amenable to 8-bit quantization, which can substantially reduce model size and latency with minimal accuracy drop. This is achieved through specific network design choices like using depthwise convolutions and training with quantization-aware techniques. 8-bit NanoDet is 4x smaller and 2-4x faster than the full-precision model with only 0.6% mAP loss.

The end result is a highly efficient detector that punches well above its weight. NanoDet-Plus, the latest version, achieves 34.1 mAP on COCO using a backbone with only 1.8M parameters. The full model is less than 2 MB in size and runs at 97 FPS on a single ARM CPU core. This is state-of-the-art accuracy for sub-5MB detectors.

Speed and Accuracy Comparison

To put NanoDet‘s performance in context, let‘s compare it to other leading lightweight detectors on the COCO benchmark. The scatter plot below shows the tradeoff between accuracy (mAP) and efficiency (millions of parameters) for different models:

[Scatter plot comparing NanoDet, YOLOX, YOLOv5, MobileNet-SSDLite, etc. on COCO mAP vs. number of parameters]

NanoDet-Plus achieves the highest mAP among sub-5M parameter models, outperforming the popular YOLOX-Nano and YOLOv5n by 1.5-2%. It is also much more parameter-efficient than MobileNet-SSDLite with comparable accuracy.

In terms of inference speed, NanoDet is one of the fastest detectors around, especially on ARM CPUs commonly used in mobile devices. The following table compares frames per second (FPS) for different models on the Snapdragon 865 CPU:

Model	mAP	FPS
NanoDet-Plus	34.1	97
YOLOX-Nano	32.6	81
YOLOv5n	31.9	75
MobileNet-SSDLite	33.8	42

NanoDet-Plus achieves the highest FPS while also having the highest accuracy. Its efficient Ghost-PAN and depthwise convolutional backbone enable fast inference even on mobile CPUs.

Training and Deployment

NanoDet is implemented in PyTorch and can be trained on custom datasets in addition to COCO. The code supports multi-scale training, random flipping, photometric distortion, and other augmentations to improve robustness. Due to the model‘s compactness, it can be trained with large batch sizes of 80+ images on a single 8GB GPU. This allows for fast iteration and experimentation.

To deploy NanoDet on mobile devices, it can be exported to the ONNX format and optimized with quantization. The quantized model can be converted to efficient mobile frameworks like TFLite, TNN, or NCNN for inference. On a typical smartphone, NanoDet runs at 20-30 FPS while drawing only 200-300 mW of power.

An end-to-end demo of NanoDet mobile deployment is available in the GitHub repo. It includes an Android app that runs the model on either CPU or GPU backends, as well as scripts for converting the model and benchmark its speed and accuracy. With the included pre-trained models, it‘s easy to get started with NanoDet and see its impressive efficiency firsthand.

The Road Ahead

NanoDet and other lightweight detectors are rapidly advancing the state-of-the-art for edge AI. However, there is still a significant gap between the best mobile models and larger server-side models – EfficientDet-D7 achieves 55.1 mAP compared to NanoDet‘s 34.1 mAP. Closing this gap further while staying within mobile resource constraints is an active area of research.

One promising direction is Neural Architecture Search (NAS), which automates the design of efficient network architectures. Rather than manually tweaking backbone and head structures, NAS can discover novel combinations of operations that optimize accuracy and latency for a given device. Several recent works like MnasFPN and SpineNet have used NAS to achieve state-of-the-art results on mobile.

Another exciting frontier is unsupervised and self-supervised learning, which can reduce the need for expensive labeled data. Techniques like MoCo, BYOL, and SwAV have shown that it‘s possible to learn good visual representations from unlabeled images alone, by training the model to be consistent across different views of the same image. These pre-trained representations can then be fine-tuned for detection with limited labeled data. Self-supervised learning could be especially valuable for mobile detectors, where labeled data is often scarce.

On the deployment side, we can expect to see more integration of detectors into end-to-end perception systems for mobile robotics, AR, and other applications. This will require detectors to not only be accurate and efficient, but also robust to real-world variations, able to run on-device without cloud support, and able to adapt to new tasks and environments over time. Building these integrated "always-on" perception systems that can evolve in the wild is a major challenge for the edge AI community.

Conclusion

NanoDet is a cutting-edge object detector that squeezes maximal accuracy and speed out of a minimal footprint. Through innovations in loss functions, label assignment, architecture design, and quantization, it achieves an impressive 34.1 mAP while being small and fast enough to run on a $50 smartphone. As AI continues to move from the cloud to the edge, NanoDet and its descendants will be essential tools for mobile developers.

Deploying NanoDet is just the first step in building mobile AI applications. Collecting good data, defining relevant tasks, integrating the model into a larger system, and testing it thoroughly in the real world are all critical challenges. But by standing on the shoulders of efficient detection giants like NanoDet, mobile AI developers can focus on these higher-level problems while still leveraging state-of-the-art perception.

Over the next few years, we can expect to see an explosion of AI-powered mobile apps that enhance our daily lives in countless ways. And behind the scenes, unsung heroes like NanoDet will be working hard to turn pixels into insight, one frame at a time.

References

• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection. Li et al., NeurIPS 2020.
• FCOS: Fully Convolutional One-Stage Object Detection. Tian et al., ICCV 2019.
• NanoDet: A Super-Fast and Accurate Object Detector for Edge Devices. Yan et al., CVPR Workshop 2021.
• NanoDet-Plus: Super Fast and High Accuracy Lightweight Face Detector. Yan et al., arXiv 2021.
• YOLOX: Exceeding YOLO Series in 2021. Ge et al., arXiv 2021.
• YOLOv4: Optimal Speed and Accuracy of Object Detection. Bochkovskiy et al., arXiv 2020.
• MobileNetV3: Searching for MobileNetV3. Howard et al., ICCV 2019.
• GhostNet: More Features from Cheap Operations. Han et al., CVPR 2020.
• Gaussian YOLO: An Efficient and Accurate Lightweight Object Detector for Edge GPUs. Choi et al., IEEE Access 2021.
• Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. Jacob et al., CVPR 2018.