Top AI and Machine Learning Conferences in 2020: Highlights, Trends, and What‘s Next

2020 was a historic year that will be remembered for the unprecedented challenges brought on by the COVID-19 pandemic. Virtually every aspect of life was impacted, including the world of academic research and conferences.

For the AI and machine learning community, the year began on a high note with several prominent conferences planned, including ICLR, CVPR, ICML, ACL, ECCV, NeurIPS, and more. However, as the scale and severity of the pandemic became apparent, conference organizers had to make quick and difficult decisions.

With global travel restrictions, lockdowns and health concerns ruling out physical gatherings, most major AI/ML conferences moved to an entirely virtual format for the first time. This required a massive logistical effort to restructure events for an online setting, from developing virtual conferencing platforms to pre-recording talks and ensuring accessibility across time zones.

But the community adapted quickly, and the resulting virtual conferences turned out to be hugely successful. In fact, many of the top conferences saw record-breaking attendance and participation, with researchers from across the globe able to join without the barriers of travel and cost.

Here are some of the key facts and figures from the top AI/ML conferences in 2020:

As the numbers show, participation skyrocketed compared to previous years. ICLR more than doubled its attendance from 2019, while NeurIPS, the largest AI conference, saw an incredible 66% increase with over 22,000 attendees logging in virtually.

But it wasn‘t just the scale of the conferences that was impressive – the quality and significance of the research presented was equally remarkable. Let‘s dive into some of the breakthroughs and trends from the major AI/ML conferences in 2020.

Natural Language Processing

NLP has been one of the hottest areas in AI recently, and this was evident at conferences like ACL, EMNLP and NeurIPS. There were significant advancements in unsupervised pre-training and transfer learning, led by models such as Google‘s BERT, Facebook‘s RoBERTa, and OpenAI‘s GPT-3.

Few-shot learning, which enables models to learn new tasks with very little training data, also saw progress with techniques like pattern-exploiting training presented at EMNLP. This could help democratize NLP and lower the barriers to building language models for new domains.

Machine translation, question answering, sentiment analysis, and dialogue systems continued to advance with new architectures and training methods. Tutorials and workshops also covered important issues like bias and fairness in NLP, model interpretability, and applications for low-resource languages.

One notable award winner at EMNLP was "Digital Voicing of Silent Speech" by David Gaddy and Dan Klein, which converts silently mouthed speech into text – a fascinating interface that could help people with speaking difficulties or allow private communication in public places.

Computer Vision

The major computer vision conferences like CVPR, ICCV and ECCV featured groundbreaking work in areas like recognition, detection, segmentation, 3D vision, multi-modal learning, and more.

There was a lot of buzz around transformer models, which have taken the NLP world by storm and are now seeing adoption in vision tasks. Several papers proposed variants of transformers customized for images and videos, such as the Vision Transformer (ViT) from Google Brain.

Self-supervised learning also gained traction as an effective paradigm to learn visual representations without requiring large labeled datasets. Papers like BYOL and SimCLR proposed new methods to learn transferable features by training models to be consistent under different augmentations of the same image.

Exciting progress was also made in 3D perception, with papers on neural scene representations, novel view synthesis, and unsupervised discovery of object categories from video. The Best Paper award at ECCV went to "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" from UC Berkeley, Google Research and UC San Diego.

Other research directions included using vision models for social good (e.g. assisting the blind), federated learning for privacy-preserving visual recognition, and new datasets and benchmarks like the Objectron dataset for 3D object detection.

Reinforcement Learning and Robotics

Reinforcement learning (RL) research was well-represented at conferences like ICML, NeurIPS and CoRL (Conference on Robot Learning). A key focus was on improving the sample efficiency of RL algorithms, as training agents in complex environments can require huge amounts of interaction data.

Several papers proposed ways to incorporate planning, world models and offline datasets into RL training. Model-based RL methods aim to learn a predictive model of the environment and use it to optimize a policy, either through planning or by generating synthetic experience.

Imitation learning also remained a popular approach, with papers on learning reward functions from human preferences, one-shot imitation from video, and imitation in partially observable settings.

In the robotics domain, there was an emphasis on learning dexterous manipulation skills, often in a self-supervised manner by allowing agents to interact with objects and learn from the resulting trajectories. Papers explored topics like contact-rich manipulation, non-prehensile manipulation (e.g. pushing), and learning from tactile feedback.

The Best Paper awards at CoRL went to "Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter" from MIT, NVIDIA and Harvard, and "MOPO: Model-based Offline Policy Optimization" from Stanford and Salesforce Research.

Fairness, Accountability and Transparency

In recent years, there has been growing recognition of the potential for bias and fairness issues in AI systems, as well as the need for transparency and accountability. Encouragingly, these topics received increased emphasis at the major ML conferences in 2020.

There were dedicated workshops on AI ethics, fairness, privacy and security at NeurIPS and ICML. Many papers also incorporated these considerations, such as analyzing bias in language models, proposing techniques for differentially private ML, and studying the implications of deepfakes.

One notable example is "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" by Emily M. Bender, Timnit Gebru et al., which critiqued the trend of ever-larger language models and argued for more thoughtful approaches that consider ethics and fairness. (Note: This paper actually appeared at FAccT 2021 but was widely discussed in 2020).

As AI systems become more influential in high-stakes domains, it‘s crucial that the research community continue to prioritize responsible development and deployment. The increased focus on these issues at top ML conferences is an encouraging sign.

The Road Ahead

While 2020‘s virtual conference format had many benefits in terms of accessibility and scale, most in the AI/ML community are understandably eager to meet again in person. As vaccines roll out and restrictions lift, a number of conferences in 2021 are considering hybrid physical-virtual formats to get the best of both worlds.

But there are also many open questions. Will we continue to see the expanded participation that virtual conferences enabled? Will there be a shift towards more focused workshops and smaller regional events? How can we build on the momentum of virtual conferences to make research more open and accessible?

Regardless of the format, I expect to see continued progress on the key challenges facing the field – from fundamental research problems like unsupervised learning and reasoning to important issues of fairness, robustness and scalability.

As an AI/ML practitioner, I‘m excited to keep engaging with the vibrant research community and to see the scientific advances from each new conference season. If 2020 is any indication, even in the midst of a global crisis, the passion and ingenuity of the AI/ML community remain strong.

To learn more, I encourage you to explore the wealth of publicly available conference content from 2020 – paper PDFs, videos, slides, posters, and more. Many conferences also have active forums for discussion and Q&A.

Here are a few places to start:

• ICLR 2020 Oral Papers
• CVPR 2020 Open Access
• NeurIPS 2020 Proceedings
• ACL 2020 Virtual Portal

Feel free to share your own highlights from the past year of AI/ML conferences in the comments. Here‘s to a brighter 2021 and to many more years of discovery and progress!

Disclosure: The author has published research papers at ICML, NeurIPS, ICLR, and EMNLP.