Baidu Shifts Self-Driving into High Gear with Release of ApolloScape, the World‘s Largest Autonomous Driving Dataset

Self-driving cars, once a science fiction dream, are rapidly becoming a reality thanks to breakthroughs in artificial intelligence and the availability of large-scale datasets for training autonomous systems. As one of the world leaders in AI, Chinese tech giant Baidu has been at the forefront of the self-driving revolution with Apollo, its open autonomous driving platform. Now, by releasing ApolloScape, the largest open-source self-driving dataset ever, Baidu is aiming to pour rocket fuel on autonomous driving research and development.

Accelerating Autonomy with Apollo

First announced in 2017, Baidu‘s Apollo platform is a comprehensive, secure, all-in-one solution that enables partners to develop their own autonomous vehicles. Apollo supports all major functions of an autonomous vehicle, providing perception, prediction, planning, and control capabilities for complex urban environments. The latest version, Apollo 7.0, features urban road autonomous driving, cloud simulation, and enhanced out-of-town scenarios.^1

Apollo‘s capabilities have been validated over millions of autonomous miles driven in challenging real-world environments. The platform has already enabled a fully driverless robotaxi service in China, making autonomous ride-hailing a reality.^2 With a thriving ecosystem of over 200 partners worldwide^1, Apollo has become one of the most influential autonomous driving forces globally.

Inside ApolloScape: A Self-Driving Data Trove

To further accelerate autonomous driving research, Baidu has open-sourced ApolloScape, a diverse, information-rich dataset that is an order of magnitude larger than any previous self-driving dataset. ApolloScape boasts over 140,000 frames of high-resolution camera images and 28,000 lidar point cloud frames, meticulously annotated with pixel-level semantic segmentation of 26 common street scene classes.^3

Just how big is ApolloScape compared to other self-driving datasets? As shown in the chart below, it dwarfs popular open datasets like Kitti and Cityscapes that have been widely used in autonomous driving research:

ApolloScape‘s scale is impressive, but it‘s the richness and diversity of the data that really sets it apart. For each scene, it provides:

• High-resolution images (3384×2710) from multiple cameras
• Dense 3D lidar point clouds (over 1M points per frame)
• Pixel-level semantic segmentation of 26 classes
• Lanemarking and localization data
• Pose information and online calibration

This multi-sensor data covers a wide variety of real-world urban driving scenarios across multiple cities in China. Baidu collected the data using a fleet of vehicles equipped with industrial-grade cameras, Velodyne 64-beam lidars, high-precision GNSS, and IMUs.^3

To create the semantic annotations, a team of trained annotators manually labeled each pixel in every image, a labor-intensive process requiring thousands of hours. The result is a pixel-perfect segmentation of common road objects like cars, pedestrians, traffic signs, buildings, and more, as illustrated here:

^4

Fueling the Future of Self-Driving Research

ApolloScape‘s vast size and rich annotations make it an unparalleled resource for advancing the state-of-the-art in self-driving perception. Modern autonomous driving systems rely heavily on deep learning models trained on huge datasets to accurately perceive and understand complex real-world scenes.

While techniques like transfer learning and sim-to-real can help, there‘s no substitute for extensive, high-quality real-world data when it comes to building robust models for the immense diversity of operational design domains self-driving vehicles must handle. The famous RAND study estimated that hundreds of millions of autonomous miles may be needed to statistically demonstrate the safety of self-driving cars, underscoring the need for large-scale datasets.^5

By releasing a free public dataset of unprecedented size and quality, Baidu aims to democratize access to data and catalyze research breakthroughs. ApolloScape dramatically lowers the barrier to entry, enabling academics and smaller companies to participate in pushing self-driving technology forward.

From an AI/ML perspective, ApolloScape unlocks a wide range of research possibilities:

• Training state-of-the-art semantic segmentation models for pixel-wise scene understanding
• 3D object detection and tracking from lidar point clouds
• Sensor fusion to exploit complementary camera-lidar data
• Freespace estimation and occupancy mapping for navigation
• Localizing in HD maps for precise positioning
• Imitation learning and end-to-end driving models
• Unsupervised domain adaptation to handle changing environments

With its diversity of scenes, ApolloScape can significantly improve model robustness and generalization. The chart below shows how Baidu‘s DeepLab semantic segmentation model performs on the ApolloScape test set compared to Cityscapes:^4

The lower relative performance on ApolloScape highlights the additional challenges posed by its greater diversity and suggests that models trained on it may be more robust. As ApolloScape matures and expands, it will support the development of even more capable and resilient self-driving systems.

Driving into the Future

ApolloScape‘s release represents a major milestone for self-driving research, but Baidu sees it as just the beginning. The company has committed to continuously expanding the dataset to cover an even broader range of scenarios, including more geographic areas, weather conditions, and driving situations.^6 Future versions will incorporate data from highway driving, rural roads, and additional cities across China.

Baidu is also exploring ways to further enrich the dataset with new annotation types like 3D bounding boxes, bird‘s eye view representations, and HD mapping features.^6 As the industry moves towards greater standardization, upcoming ApolloScape releases will align with common data formats to maximize compatibility and utility for both academic and commercial researchers.

As one of the leading forces in the autonomous driving revolution, Baidu has a vested interest in accelerating the pace of self-driving research and deployments industry-wide. According to a recent McKinsey report, China and the US are leading the global autonomous vehicle race, with China expected to see mass deployment of L4 self-driving between 2025-2030, generating trillions in economic value.^7

By catalyzing an ecosystem of researchers and partners through open platforms like Apollo and public datasets like ApolloScape, Baidu stands to benefit from the resulting pace of innovation. Large-scale multi-sensor datasets are essential infrastructure for developing production-quality self-driving systems. Sharing this valuable data is a rising tide that can lift all boats.

Looking ahead, there are many exciting directions for the future evolution of self-driving datasets. Active learning techniques could automatically mine large unlabeled datasets for examples that would be most informative to annotate.^8 Synthetic data rendering and domain randomization could augment real datasets with diverse simulated scenarios.^9 With the advent of 5G networks enabling smart cities and vehicle-to-everything (V2X) communication, real-time crowdsourced datasets from fleets of sensor-equipped vehicles may one day emerge.^10

In the meantime, ApolloScape stands as the largest and most comprehensive self-driving dataset available to public researchers. As academics and companies leverage this rich new resource to fuel AI breakthroughs, the dream of safe, accessible autonomous transportation for all will keep accelerating towards reality. Baidu‘s groundbreaking release is poised to go down in history as a key milestone that helped drive the world forward to the coming age of autonomy.