# Facebook Accelerates NLP and Navigation Research with New Open Source Datasets

- Canonical: https://33rdsquare.com/facebook-open-sources-dataset-on-nlp-and-navigation-every-data-scientist-should-download/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

Facebook‘s AI research lab (FAIR) has long been at the forefront of open sourcing high-quality datasets and models to spur progress in machine learning. In domains from computer vision to speech recognition, FAIR‘s open source initiatives have become go-to resources for academics and industry practitioners alike.

Recently, Facebook has turned its attention to natural language processing (NLP) and navigation/spatial reasoning as key areas for AI advancement. The explosive growth of digital assistants, robotics, autonomous vehicles, and AR/VR has made teaching machines to understand and navigate the world a major priority.

To accelerate progress, Facebook has released several cutting-edge open source datasets targeting these challenges. Most notably, the Talk the Walk dataset, created in partnership with the University of Montreal, aims to bridge the gap between natural language and visual navigation.

## Why Open Source Datasets Matter

In the early days of modern AI, researchers had to rely on small, proprietary datasets to train and benchmark new models. Collecting large-scale, high-quality datasets was prohibitively time-consuming and expensive for all but the largest tech companies.

The open source revolution in AI changed that. Initiatives like [ImageNet](http://www.image-net.org/), [COCO](https://cocodataset.org/), and [LibriSpeech](https://www.openslr.org/12) made vast amounts of training data available to all. This democratized access and sparked an explosion of new research from labs around the world.

According to the [2022 AI Index Report](https://aiindex.stanford.edu/report/), the number of open source AI software libraries grew nearly 7x from 2018 to 2022. Membership in online AI research communities like Papers With Code has skyrocketed. This collaborative, open ecosystem is now the primary engine driving AI/ML innovation.

For a company like Facebook, there are clear business incentives to contribute to the open source ecosystem. Releasing a popular dataset establishes thought leadership, attracts research talent, and ensures new models and architectures are compatible with Facebook‘s platforms and use cases. Altruistically, it also aligns with the company‘s [stated mission](https://about.facebook.com/company-info/) to "give people the power to build community and bring the world closer together."

## Introducing Talk the Walk

First released in 2018, Talk the Walk is one of Facebook‘s most ambitious open source datasets to date. It contains three key components:

1. Highly detailed 2D maps of sections of New York City (from OpenStreetMap)
2. 360-degree panorama images of locations on those maps (from Google Street View)
3. Over 10,000 natural, written conversations between humans providing navigation instructions

The goal of Talk the Walk is to train AI agents that can understand natural language instructions and navigate accordingly in a realistic virtual environment. It‘s a task that requires deep linguistic, visual, and spatial reasoning skills.

Here‘s a simplified example of how it works: the AI agent (called the "tourist") is dropped into a random location in the virtual NYC environment. It can "look around" by accessing the 360-degree Street View panoramas.

Meanwhile, a human (the "guide") has access to a 2D map showing the tourist‘s location. The guide provides written, natural language instructions to navigate the tourist to a target location. Neither the guide nor the tourist has access to the other‘s view.

After each instruction, the tourist must choose to move forward, turn left/right, or indicate it has reached the goal. The conversation continues until the tourist reaches the target (success) or exhausts the maximum number of dialog turns (failure).

Here‘s a real dialog from the dataset:

![Example dialog from Talk the Walk dataset](https://33rdsquare.com/talk-the-walk-dialog-example.jpg)

This simple setup belies an extremely challenging machine learning task. To succeed, the AI tourist must learn to:

- Interpret high-level natural language instructions (e.g. "turn left at the next intersection")
- Ground those instructions to visual observations of the environment
- Reason about spatial relationships and navigate accordingly
- Engage in multi-turn dialog to clarify ambiguous directions or request more info

It‘s a far cry from typical NLP tasks like sentiment analysis or named entity recognition. Talk the Walk forces the AI to connect language to the real world, an essential skill for any robot or digital assistant.

## Dataset Specs and Baseline Results

The full Talk the Walk dataset contains 10,714 human-written navigation dialogs, totaling over 5 million words. This includes 4,500 unique starting locations covering an area of NYC approximately 5km x 3km.

Each dialog consists of up to 8 pairs of guide instructions and tourist actions (dialog turns). The median dialog length is 5 turns. In total, there are over 37,000 unique guide instructions and 227,000 tourist actions.

Here‘s a breakdown of the most common tourist actions:

| Action | Frequency |
| --- | --- |
| Forward | 40.5% |
| Turn Left | 20.3% |
| Turn Right | 20.1% |
| Stop (goal) | 19.1% |

Impressively, the human guides are able to successfully navigate the tourist to the goal location in 80.6% of dialogs using only natural language instructions.

As a benchmark, Facebook trained several seq2seq neural models to play the role of the tourist. The best performing model used a ResNet image encoder and Transformer text encoder/decoder. After fine-tuning, this model achieved a successful navigation rate of 12.8%, well below human performance.

Here are the full baseline results from the [Talk the Walk paper](https://arxiv.org/abs/1807.03367):

![Baseline model results on Talk the Walk dataset](https://33rdsquare.com/talk-the-walk-baseline-results.jpg)

These results underscore the difficulty of the task and the room for improvement. Even with state-of-the-art deep learning architectures, grounding language in complex visual environments remains an open challenge.

## Comparison to Other Navigation Datasets

Talk the Walk is not the only open source dataset targeting vision-and-language navigation (VLN). Other notable datasets in this area include:

- [Room-to-Room (R2R)](https://openaccess.thecvf.com/content_cvpr_2018/papers/Anderson_Vision-and-Language_Navigation_Interpreting_CVPR_2018_paper.pdf): Navigation in 3D home environments using human-written instructions
- [Touchdown](https://arxiv.org/abs/1811.12354): Navigation in real-world urban environments using instructions written from Google Street View
- [ASKNAV](https://arxiv.org/abs/2104.07662): Indoor navigation via question-answering and visual observations

While these datasets share the same high-level goal as Talk the Walk (teaching agents to follow natural language navigation instructions), there are a few key differences that make Talk the Walk unique:

1. Talk the Walk uses an interactive, dialog-based setup, while most other datasets use a single set of instructions to navigate from start to finish. This makes Talk the Walk more challenging but also more natural.
2. Talk the Walk is one of the only VLN datasets situated in real-world urban environments. Most others use synthetic 3D environments or highly constrained real-world interiors.
3. Talk the Walk has the most detailed map data, thanks to the use of OpenStreetMap. This allows for grounding instructions to named entities like streets, businesses, and landmarks.
4. With over 10k dialogs, Talk the Walk is one of the largest VLN datasets available. Only R2R (21k episodes) and Touchdown (9.3k episodes) come close in terms of scale.

Of course, there are also areas where Talk the Walk lags behind other datasets. The Street View imagery is high resolution but static, lacking the multi-modal sensing of a true embodied agent. It also covers a limited area of NYC, while datasets like StreetLearn span multiple cities.

Overall though, Talk the Walk remains one of the most challenging and comprehensive resources for research at the intersection of language, vision, and navigation. Its unique dialog-based setup in particular opens up promising avenues for human-robot interaction.

## Applications and Future Potential

Models trained on the Talk the Walk dataset have myriad potential applications. Some of the most exciting include:

- **Robotics**: Enabling robots to interpret natural language commands and navigate complex environments, from homes to warehouses to city streets. Could power everything from elderly care bots to last-mile delivery drones.
- **Digital Assistants**: Improving the spatial reasoning and navigational intelligence of AI assistants like Alexa, Siri, and Google Maps. Imagine an assistant that can guide you step-by-step to your destination, adapting to your feedback and questions along the way.
- **Gaming and Simulation**: Powering more realistic NPC behavior and dialog in video games and virtual worlds. Could also enhance sim-to-real transfer learning for robot navigation.
- **Accessibility**: Helping visually impaired individuals safely navigate urban environments using only voice commands and minimal additional sensing.

Beyond these applications, Talk the Walk is a powerful testbed for advancing core AI capabilities in areas like commonsense reasoning, contextual language understanding, and human-AI collaboration. It‘s a prime example of why benchmarks need to move beyond narrow, siloed tasks to more open-ended challenges grounded in the real world.

As the authors note in the original [Talk the Walk blog post](https://ai.facebook.com/blog/talk-the-walk-teaching-ai-to-navigate-cities-by-talking-to-it/), "This work is part of a larger program aimed at developing intelligent systems that can intuitively communicate with humans…We believe that the path to creating truly intelligent systems is to allow them to learn by engaging with humans in realistic settings."

Of course, realizing this potential will require substantial additional breakthroughs. The gap between the best performing Talk the Walk model (12.8% success rate) and human performance (80.6%) is still vast. We‘ll need innovations in multi-modal learning, sample efficiency, unsupervised pre-training, and other areas to close it.

There are also important responsible AI considerations to keep in mind with a dataset like Talk the Walk. The human dialogs are crowd-sourced, opening the door to biases around gender, race, age, and ability in the instructions. The image data may also encode biases based on the demographics of the neighborhoods captured. Researchers will need to audit their models carefully to ensure equitable performance.

Privacy is another concern when dealing with street-level imagery, even with blurring applied. As the collection and use of public datasets like these increases, the AI community will need to remain vigilant about protecting individual rights.

## Getting Started with Talk the Walk

If you‘re a data scientist or ML researcher looking to dive into NLP and navigation, Talk the Walk is a great place to start. Here‘s a quick guide to getting up and running with the dataset:

1. Download the data: The full Talk the Walk dataset is available for download from the [official GitHub repo](https://github.com/facebookresearch/talkthewalk). You‘ll need around 32 GB of free disk space.
2. Explore the data: Take some time to familiarize yourself with the dataset structure and contents. The repo includes detailed documentation on the file formats, naming conventions, and data fields. Visualize some of the dialogs and maps to get a sense of the task.
3. Set up your environment: Talk the Walk is compatible with PyTorch, a popular open source deep learning library. You‘ll want to create a fresh Python virtual environment and install PyTorch, torchvision, and a few other dependencies listed in the repo.
4. Run the baselines: The GitHub repo includes code for training and evaluating the baseline seq2seq models discussed in the paper. Running these is a great way to validate your setup and generate some initial results. You can then experiment with different architectures and hyperparameters.
5. Explore related resources: In addition to the main dataset, Facebook has also released pre-trained ResNet embeddings of the Street View images and open source implementations of relevant models like the [Transformer](https://github.com/facebookresearch/vilbert-multi-task). The [Habitat](https://aihabitat.org/) and [TouchDown](https://github.com/lil-lab/touchdown) platforms also provide helpful tools for navigation research.
6. Contribute to the community: As you work with Talk the Walk, consider sharing your code, models, and results with the broader research community. Platforms like Papers With Code and Weights & Biases make it easy to collaborate and compare progress on open datasets like this.

Above all, remember that there are still many open questions and opportunities for innovation in this area. Don‘t be afraid to experiment with novel architectures, training techniques, and problem formulations. The next big breakthrough in AI navigation could be just around the corner.

## The Road Ahead

Datasets like Talk the Walk offer a tantalizing glimpse of a future where intelligent agents can understand and navigate the world as intuitively as humans do. But realizing that future will require a concerted effort from researchers, engineers, and ethicists across academia and industry.

We‘ll need to develop more efficient algorithms that can learn from small amounts of interaction, like humans do. We‘ll need more open source datasets that capture the full diversity of real-world environments and behaviors. And we‘ll need robust frameworks for auditing and mitigating the potentially harmful biases in our navigation models.

None of these challenges are insurmountable. With initiatives like Talk the Walk, Facebook and other AI leaders are laying the groundwork for a new era of grounded, multimodal, interactive AI. It‘s now up to the rest of the research community to pick up the torch and carry it forward.

As the great computer scientist Alan Kay once said, "The best way to predict the future is to invent it." With datasets like Talk the Walk, we now have the tools to invent a future where intelligent agents are not just servants, but partners in our daily lives. Let‘s get started.

---

Source: [Facebook Accelerates NLP and Navigation Research with New Open Source Datasets](https://33rdsquare.com/facebook-open-sources-dataset-on-nlp-and-navigation-every-data-scientist-should-download/)