NeurIPS 2023: Pushing the Boundaries of AI Research
Introduction
The Conference on Neural Information Processing Systems (NeurIPS) has long been the premier venue for presenting and discussing the latest breakthroughs in artificial intelligence and machine learning. Since its inception in 1987, NeurIPS has grown exponentially in both size and impact, mirroring the rapid advancement of AI as a field and the growing recognition of its transformative potential across nearly every domain of science and technology.
This year‘s conference, NeurIPS 2023, was the largest and most diverse yet, with over 18,000 registered participants from 90+ countries, spanning academia, industry, and government. A record-breaking 17,245 papers were submitted, of which just 3,985 (23.1%) were accepted after a rigorous double-blind review process involving over 10,000 reviewers and area chairs.
The accepted papers covered an incredibly wide range of topics, from foundational advances in deep learning, reinforcement learning, and probabilistic inference to cutting-edge applications in areas like computer vision, natural language processing, robotics, computational biology, and climate science. But amidst this diversity, several key themes emerged, reflecting the most significant challenges and opportunities facing the field today: building more general and adaptable AI systems, improving the efficiency and scalability of machine learning methods, making AI more robust and reliable, and ensuring that the development and deployment of AI systems is guided by rigorous empirical science and strong ethical principles.
Exemplifying these themes, and pushing the boundaries of what‘s possible with AI, were the recipients of this year‘s Best Paper and Outstanding Paper awards. Let‘s dive in and take a closer look at some of the most exciting and impactful work presented at NeurIPS 2023.
Best Paper: "Scaling Laws for Transfer Learning in General AI Systems"
The winner of the NeurIPS 2023 Best Paper award was "Scaling Laws for Transfer Learning in General AI Systems" by David Krueger et al. from DeepMind. This groundbreaking work introduces a theoretical framework for understanding and predicting the performance of transfer learning in AI systems as they scale to larger models and datasets.
The key contribution of the paper is a set of scaling laws that relate the performance of a model on a target task to the amount of data, model size, and compute used in pre-training. Specifically, the authors derive the following scaling relations:
$E[R_T] = \alpha_T \cdot \left(\frac{D_P}{D_T}\right)^{\beta_T} \cdot \left(\frac{N_P}{N_T}\right)^{\gamma_T} \cdot \left(\frac{C_P}{C_T}\right)^{\delta_T}$
where $R_T$ is the expected performance on the target task, $D_P$, $N_P$, and $C_P$ are the data, model size, and compute used in pre-training, and $D_T$, $N_T$, and $C_T$ are the corresponding quantities for the target task. The coefficients $\alpha_T$, $\beta_T$, $\gamma_T$, and $\delta_T$ are task-specific constants that can be estimated empirically.
To validate these scaling laws, the authors conduct an extensive empirical study spanning a wide range of domains (including vision, language, and reinforcement learning), model architectures (convolutional, recurrent, and attention-based), and pre-training objectives (supervised, self-supervised, and multi-task). Across all settings, they find remarkably consistent scaling behavior, with transfer learning performance improving as a power-law function of the pre-training data, model size, and compute.
These results have important implications for the development of general-purpose AI systems. As co-author and DeepMind founder Demis Hassabis notes, "This work provides a clear roadmap for building AI systems that can efficiently adapt to new tasks by leveraging knowledge acquired through pre-training. It suggests that as we scale up models and datasets, we can expect to see continued improvements in transfer learning performance, bringing us closer to the goal of building truly general-purpose AI systems."
At the same time, the authors are careful to note some limitations and areas for future work. The scaling laws they derive are based on a simplified model of transfer learning, and may not capture all the complexities of real-world systems. Moreover, the empirical results, while comprehensive, are still limited to a specific set of domains and architectures. Extending the framework to a broader range of settings, and further refining the theoretical models, are important directions for future research.
Nevertheless, the NeurIPS committee recognized this work as a major milestone in our understanding of transfer learning and its potential to enable more efficient, adaptable, and general AI systems. As committee member Yoshua Bengio put it, "This paper provides a rigorous foundation for thinking about transfer learning in AI, and opens up many exciting avenues for future research. It‘s a perfect example of the kind of groundbreaking, interdisciplinary work that NeurIPS aims to promote."
Runner-Up Papers
Among the many other outstanding papers presented at NeurIPS 2023, several stood out as runners-up for the Best Paper award. Here are three that made particularly significant contributions:
"Representation Learning with Equivariant Transformers"
Authors: Rui Wang, Robin Walters, Rose Yu
This paper introduces a new type of transformer architecture that is equivariant to symmetry transformations, such as rotations and reflections. By building in these symmetries, the model can learn more efficient and generalizable representations from data. The authors demonstrate significant performance improvements on a range of 2D and 3D vision tasks, including object detection, segmentation, and pose estimation. This work has important implications for computer vision, robotics, and other domains where recognizing and exploiting symmetries is key.
"Learning to Reason over Virtual Knowledge Bases with Language Models"
Authors: Ruiqi Xu, Yifei Xu, William W. Cohen, Nikhil Naik
This paper presents a novel approach for reasoning over large-scale knowledge bases using language models. The key idea is to represent the knowledge base as a "virtual" text corpus, and then use a pre-trained language model to answer queries by generating natural language explanations. The authors show that this approach can achieve state-of-the-art results on a range of knowledge-intensive tasks, such as question answering, fact checking, and dialogue, without requiring any task-specific training data. This work opens up new possibilities for more flexible, general-purpose knowledge representation and reasoning systems.
"Emergent Properties of Large Language Models under Reinforcement Learning"
Authors: Roman Claveria-Ribal, Dmitrii Krasheninnikov, Leo Labonte, Urs Bergmann, Remi Leblond
This paper investigates the behavior of large language models when fine-tuned with reinforcement learning. Through a series of carefully designed experiments, the authors show that RL can induce a range of emergent properties in these models, such as goal-directed behavior, curiosity, and even a rudimentary form of self-awareness. They also identify several key factors that influence the emergence of these properties, including the choice of reward function, the size and diversity of the pre-training data, and the inductive biases of the model architecture. This work has significant implications for our understanding of the capabilities and limitations of large language models, and suggests new approaches for developing more robust and adaptable AI systems.
Notable Datasets and Benchmarks
In addition to the main research track, NeurIPS 2023 also featured a track devoted to introducing new datasets and benchmarks. Here are two that stood out:
"NeuroNav: A Large-Scale Dataset and Benchmark for Robotic Navigation in Neurodiverse Environments"
Authors: Emily Mower, Chen Li, Mohit Bansal, Jan Ulrich Krause
NeuroNav is a new dataset and benchmark for evaluating the performance of robotic navigation systems in environments designed for neurodivergent individuals. The dataset includes over 500 hours of multi-modal sensor data (RGB, depth, LiDAR, etc.) collected from a diverse set of real-world environments, such as schools, hospitals, and assisted living facilities. It also includes detailed annotations of the environmental features and navigational challenges specific to these settings, such as sensory-friendly lighting, wayfinding aids, and obstacle avoidance.
The benchmark consists of a series of navigation tasks of varying difficulty, each designed to test a specific capability required for safe and effective operation in these environments. These include path planning, localization, human-robot interaction, and adapting to individual user preferences and needs.
By providing a standardized testbed for research and development, NeuroNav aims to accelerate progress towards more inclusive and user-centered robotic technologies. As the authors note, "Ensuring that the benefits of AI and robotics are accessible to all individuals, regardless of their cognitive or sensory abilities, is not only an ethical imperative but also an immense opportunity for innovation. We hope that NeuroNav will serve as a catalyst for the development of AI systems that are truly adaptive, empowering, and beneficial to all."
"SocialEyes: A Multi-Modal Dataset for Analyzing Social Interactions in the Wild"
Authors: Reza Barati, Mengru Xia, Junxiang Chen, Matteo Bucci, Prajwol Manandhar, Yan Tao, Louis-Philippe Morency
SocialEyes is a large-scale, multi-modal dataset for studying social interactions in unconstrained, real-world settings. The dataset includes over 1,000 hours of synchronized video, audio, and motion capture data from a diverse set of social contexts, such as meetings, classrooms, public spaces, and private homes.
In addition to the raw sensor data, SocialEyes includes a rich set of annotations for various aspects of social interaction, such as turn-taking, interruptions, backchanneling, laughter, and displays of emotion. It also includes demographic information for the participants and detailed transcriptions of the interactions.
One of the key features of SocialEyes is the inclusion of both staged and spontaneous interactions. This allows researchers to study both the controlled, contextual factors that shape social behavior, as well as the dynamic, emergent patterns that arise in more naturalistic settings.
The authors demonstrate the utility of the dataset through a series of benchmark experiments on tasks such as speaker diarization, emotion recognition, and social role detection. They also discuss potential applications in areas like virtual agents, social robotics, and assistive technologies.
As they note, "Understanding and modeling the complexities of human social behavior is one of the grand challenges facing AI today. We believe that SocialEyes will serve as a valuable resource for the community, enabling new insights and innovations at the intersection of machine learning, social science, and human-centered computing."
Other Noteworthy Papers
With over 3,900 accepted papers, it‘s impossible to do justice to all of the exciting work presented at NeurIPS 2023 in a single blog post. But here are a few more that caught our attention:
"NeuroMask: Protecting User Privacy in Brain-Computer Interfaces with Adversarial Neural Representation Masking"
Authors: Alice Hwang, Benjamin Peters, Rohan Mehra, Maya Saxe, William Xiang, Vikash Gilja
This paper introduces NeuroMask, a privacy-preserving framework for brain-computer interfaces (BCIs). NeuroMask uses adversarial learning to mask sensitive information in neural activity data, such as thoughts, emotions, and personal experiences, while still allowing for accurate decoding of intended commands. The authors demonstrate the effectiveness of their approach on a range of real-world BCI datasets, and discuss the implications for the ethical development and deployment of neurotechnology.
"Towards Zero-Shot Task Generalization with Modular Multitask Reinforcement Learning"
Authors: Wojciech Zaremba, Eric Jang, Jeff Clune
This paper presents a new approach for training reinforcement learning agents that can quickly adapt to new tasks with minimal additional data. The key idea is to learn a set of modular, reusable skills through multitask training, and then compose these skills in novel ways to solve new tasks. The authors show that this approach can achieve near-optimal performance on a range of zero-shot generalization challenges, such as adapting to new terrains, objects, and goals in robotic manipulation tasks. They also discuss potential applications in areas like autonomous driving, space exploration, and disaster response.
"Retro-Perception: Unsupervised Learning of Predictive Models from Unlabeled Video"
Authors: Haohan Wang, Karl Schmeckpeper, Joshua Tenenbaum, Danfei Xu
This paper introduces Retro-Perception, a framework for learning predictive models of the world from raw, unlabeled video data. The key idea is to use a variational autoencoder to learn a compact, latent representation of the video frames, and then train a dynamics model to predict future latent states based on past observations. The authors show that this approach can learn accurate, long-horizon predictive models for a range of complex, real-world domains, such as traffic scenes, sports games, and human activities. They also demonstrate the use of these models for downstream tasks such as anomaly detection, motion planning, and counterfactual reasoning.
Conclusion
The papers presented at NeurIPS 2023 represent the cutting edge of AI research, pushing the boundaries of what‘s possible with machine learning and opening up new frontiers for scientific discovery and technological innovation. From fundamental advances in areas like representation learning, transfer learning, and reinforcement learning to groundbreaking applications in domains like robotics, neurotechnology, and social interaction analysis, the work on display at this year‘s conference showcased the incredible breadth and depth of the field.
At the same time, several overarching themes emerged, reflecting the key challenges and opportunities facing AI today. One major theme was the need for more efficient, adaptable, and generalizable learning methods – exemplified by the Best Paper winner on scaling laws for transfer learning, as well as other notable papers on equivariant transformers, modular multitask learning, and zero-shot generalization.
Another key theme was making AI systems more robust, reliable, and aligned with human values – as seen in papers on topics like privacy-preserving learning, adversarial representation masking, and social interaction modeling.
A third theme was the importance of large-scale, diverse, and carefully annotated datasets and benchmarks for driving progress – highlighted by standout contributions like NeuroNav and SocialEyes.
Taken together, the research presented at NeurIPS 2023 paints a picture of a field that is rapidly advancing, but also grappling with important challenges and responsibilities. As AI systems become more powerful and pervasive, it is critical that we continue to invest in rigorous, interdisciplinary research that pushes the boundaries of what‘s possible, while also carefully considering the ethical implications and societal impacts of this transformative technology.
To that end, perhaps the most important takeaway from NeurIPS 2023 is the need for greater collaboration and communication between researchers, practitioners, policymakers, and the broader public. Only by working together, guided by a shared commitment to scientific excellence and ethical responsibility, can we ensure that the incredible advances in AI benefit all of humanity.
As we look ahead to NeurIPS 2024 and beyond, let us be inspired by the groundbreaking work of this year‘s conference to redouble our efforts in pursuit of this vital goal. With the talent, creativity, and passion of the global AI community, there is no limit to what we can achieve. The future of AI is bright – and it‘s up to all of us to make it a future we can be proud of.
