Berkeley Open Sources World‘s Largest Self-Driving Dataset: A Must-Download for Data Scientists

Self-driving vehicles have the potential to transform transportation as we know it, promising improved safety, accessibility, and efficiency. However, building autonomous driving systems is an immense challenge, requiring advanced AI and computer vision capabilities. One of the key ingredients is vast amounts of high-quality training data that captures the wide range of real-world scenarios self-driving cars must navigate.

Researchers at UC Berkeley have aimed to meet this need by releasing the world‘s largest and most diverse self-driving dataset to date. The Berkeley DeepDrive (BDD100K) dataset contains over 100,000 videos of real driving, totaling 1,100 hours of footage. Released as open data, it provides a massive and much-needed resource for the self-driving research community.

Inside the BDD100K Dataset

So what exactly does the BDD dataset contain? The core of the dataset is 100,000 videos, each about 40 seconds long at 30 frames per second and 720p resolution. This is equivalent to roughly 120 million images and over an 8th of a petabyte of raw data.

Alongside the videos, the dataset provides extensive annotations for key objects and conditions. Using a combination of machine learning and human labeling, every 10th video frame is annotated with:

• Bounding boxes for 10 object categories (e.g. cars, traffic signs, pedestrians)
• Pixel-wise drivable area maps
• Lane markings distinguishing over 10 lane marking types
• Full-sentence descriptions of each image

GPS data from smartphones accompanies the video, providing location information to map out driving trajectories across a variety of environments. Spanning bustling city streets, highways, rural roads, and residential areas, the dataset captures an extensive range of driving scenarios. Importantly, it also includes diverse weather conditions like sunny, rainy, and foggy, as well as day and night footage.

Unmatched Scale and Diversity

What truly sets the BDD dataset apart is its unprecedented scale. To put the 100,000 videos in perspective, it is over 800 times larger than the pioneering Baidu ApolloScape dataset released in 2018. Compared to benchmarks like Cityscapes and KITTI, BDD is a staggering 8,000 times bigger in terms of frames.

But beyond sheer size, BDD stands out for its rich diversity. Covering various locations across the US, it reflects a wide array of road types, traffic conditions, signage, and driving cultures. By spanning multiple weather and lighting conditions, it pushes self-driving systems to be more robust.

Together, the scale and diversity make BDD the most comprehensive public dataset to date for self-driving research. It provides an unparalleled resource for training data-hungry deep learning models on tasks like object detection, semantic segmentation, imitation learning, and more.

Accelerating Self-Driving Research

The release of the BDD dataset is a major milestone for self-driving research. Prior to this, most publicly available datasets were quite small in scale and limited in scope. This forced many researchers to rely on proprietary datasets, making it difficult to compare approaches and limiting collaboration.

With BDD, researchers around the world now have access to a dataset of a caliber previously only available to select industry labs. By creating a more level playing field, it can help accelerate progress in self-driving AI.

Some key research problems that BDD enables include:

• Pushing the boundaries of 2D/3D object detection in complex driving scenes
• Advancing road/lane segmentation that can handle challenging conditions
• Developing better trajectory prediction models for anticipating agent behavior
• Exploring online adaptation methods to handle domain shifts in self-driving

UC Berkeley has also teamed up with leading AI conferences like CVPR to organize challenges using the BDD dataset. These provide venues for researchers to benchmark new models and learn from each other‘s approaches.

Applications Beyond Autonomous Driving

While self-driving is the primary motivation for BDD, the dataset enables many other applications as well. It actually provides a great multi-purpose resource for general AI/computer vision research.

For example, with over 100,000 pedestrian labels, BDD could be incredibly valuable for research on pedestrian detection – a key problem for making roads safer. The extensive lane marking labels and drivable area maps are useful for assistive technologies like lane departure warnings. And the rich GPS data could support research in mobility patterns and navigation.

Since the videos essentially provide a snapshot of cities across the US, there may also be interesting applications in urban planning, infrastructure monitoring, and more. By releasing the dataset publicly, the potential use cases are boundless.

Challenges and Opportunities

As with any dataset, there are limitations and areas for improvement with BDD. While it covers a good range of locations and conditions, there are many complex long-tail scenarios not fully represented, like snowy weather or dirt roads. Expanding coverage of edge cases could make the dataset even more valuable.

Additionally, while the 2D box annotations are quite extensive, the dataset does not include full 3D bounding boxes. Supplementing with 3D annotations could open up research opportunities around 3D object detection and tracking.

Nonetheless, BDD provides an incredible starting point and foundation to build upon. By open sourcing the dataset, UC Berkeley has also created opportunities for the community to contribute and improve it over time. For example, researchers could collectively annotate additional attributes or conditions to make the dataset even richer.

Getting Started with BDD

For data scientists and ML engineers looking to get hands-on with self-driving research, the BDD dataset is an ideal place to start. UC Berkeley has made it available for anyone to download for free.

To make it easier to work with, the BDD team has provided the data in accessible formats like COCO JSON annotation files. They‘ve also released a set of tools and scripts for common tasks like loading annotations and visualizing object labels.

For those looking to benchmark model performance, a number of challenges have been organized around BDD. These provide standard training/validation splits and evaluation metrics for problems like object detection and drivable area segmentation. Participation is a great way to compare approaches and collaborate with other researchers.

Looking Forward

The BDD dataset is part of a larger trend towards more open and collaborative self-driving research. In the past few years, we‘ve seen a growing number of datasets, benchmarks, and open source tools aimed at accelerating progress as an AI community.

With simulation platforms like CARLA and LGSVL becoming more sophisticated, we may see future datasets that blend real-world and synthetic data at even greater scales. Techniques like active learning could help guide data collection to focus on under-represented scenarios.

As self-supervised learning methods continue to improve, we may also see more ways to leverage datasets like BDD without requiring extensive manual labeling. Representation learning on unlabeled driving video could help bootstrap object detectors and other perception models.

Ultimately, the hope is that large, diverse, and open datasets like BDD help spur innovation towards making self-driving a safe and scalable reality. While there are certainly challenges ahead, the future is bright – and data will light the way.