Detectron: Facebook‘s Powerful Open Platform for Object Detection Research

Object detection is one of the most impactful areas of artificial intelligence research today. The ability to automatically localize and classify objects in images and video has powered breakthroughs in everything from facial recognition to self-driving cars to cancer diagnosis. However, developing high-performing object detection systems has traditionally required extensive time and resources.

In January 2018, Facebook AI Research (FAIR) took a major step towards democratizing cutting-edge object detection technology with the release of Detectron. Detectron is an open-source software platform that implements state-of-the-art algorithms for object detection and segmentation, making them easier than ever to apply to real-world problems.

At its core, Detectron provides a high-quality, flexible codebase for object detection written in Python. It‘s built on top of the popular Caffe2 deep learning framework, which Facebook has invested heavily in optimizing for both cloud and mobile deployment.

Out of the box, Detectron includes implementations of today‘s most successful object detection algorithms, including:

• Mask R-CNN: Jointly performs object detection and instance segmentation, precisely outlining each distinct object
• RetinaNet: Uses a novel "focal loss" to effectively train on datasets with extreme class imbalance
• Faster R-CNN: Two-stage detector that first proposes regions of interest, then classifies and refines them
• RPN (Region Proposal Network): Efficiently generates region proposals to be fed into Faster R-CNN
• Fast R-CNN: Earlier two-stage approach that set the groundwork for Faster R-CNN
• R-FCN (Region-based Fully Convolutional Networks): Fully convolutional architecture that‘s more efficient than Faster R-CNN

What really sets Detectron apart is the sheer amount of pre-trained models included in the library. Facebook has released over 70 models spanning different detector architectures, backbone networks, and training datasets. Many of these achieve state-of-the-art results on academic benchmarks like COCO and PASCAL VOC. This rich model zoo makes it simpler than ever to apply object detection to your own dataset or integrate it into an application.

So what kinds of things can you build with Detectron? The possibilities are incredibly broad. At a recent F8 conference, Facebook CTO Mike Schroepfer showcased an app that could automatically describe the content of images for visually impaired users, powered by Detectron under the hood. Object detection is also a key component of systems like Amazon Go that enable shoppers to simply grab items and go without scanning or checking out. In the medical domain, algorithms like Mask R-CNN can help radiologists localize tumors and other abnormalities in CT scans and X-rays.

When Detectron was first released in 2018, it quickly gained traction in the research community as a go-to framework for cutting-edge object detection work. Since then, Facebook has continued to refine the platform, adding support for more recent architectures like RetinaNet along with detailed benchmarking tools.

According to Ross Girshick, research scientist at FAIR and co-creator of Detectron, the goal is to provide "a flexible and extensible object detection system that enables rapid exploration of novel research ideas." By open-sourcing the technology, Facebook aims to accelerate progress in object detection and its many applications.

To see Detectron in action, let‘s walk through a quick example of how you can use a pre-trained model for inference on a new image. After installing the Detectron library, you can load a model with just a few lines of Python:

from detectron.core.config import assert_and_infer_cfg
from detectron.core.config import cfg
from detectron.core.config import merge_cfg_from_file
from detectron.utils.io import cache_url
from detectron.utils.vis import vis_one_image
import cv2
import matplotlib.pyplot as plt
cfg_file = "configs/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml"
cfg_url = cache_url(cfg_file, use_cache=False)
merge_cfg_from_file(cfg_url)
assert_and_infer_cfg()
cfg.TEST.WEIGHTS = cache_url("https://dl.fbaipublicfiles.com/detectron/35861858/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml.08_34_42.MUo9jWBb/output/train/coco_2014_train%3Acoco_2014_valminusminival/generalized_rcnn/model_final.pkl", use_cache=False)

This code snippet loads a Mask R-CNN model pre-trained on the COCO dataset. We can then run the model on a new image and visualize the detected objects:

im = cv2.imread("input.jpg")
cls_boxes, cls_segms, cls_keyps = im_detect_all(model, im)
vis_one_image(
im[:, :, ::-1],
"output",
cls_boxes,
cls_segms,
cls_keyps,
dataset=dummy_coco_dataset,
box_alpha=0.3,
show_class=True,
thresh=0.7,
kp_thresh=2
)

And there you have it – with just a few lines of code, we‘ve leveraged a powerful object detection model to understand the contents of an image. This is just a small taste of what‘s possible with Detectron. I encourage you to dive into the Detectron Github repository and try it out for yourself!

Looking forward, object detection technology still has room for improvement, especially when it comes to recognizing objects in challenging conditions like occlusion, poor lighting, and clutter. Unsupervised and few-shot learning approaches could also help expand object detection to more niche concepts while requiring less training data.

As Piotr Dollar, research scientist at FAIR and pioneer of object detection put it, "In some sense, [object detection] is a solved problem, but in many ways it‘s absolutely not." With platforms like Detectron helping to accelerate research, the coming years are sure to bring more exciting progress. Stay tuned!