Decoding the Top Research from ICLR 2019: How Neural Networks are Advancing Machine Learning
Introduction to ICLR
The International Conference on Learning Representations (ICLR) is one of the leading academic conferences focused on deep learning and representation learning. Held annually, it brings together experts from academia and industry to share cutting-edge research and exchange ideas on topics like neural networks, optimization, unsupervised learning, reinforcement learning, and more.
ICLR uses an open peer review process and has a strong reputation for high-quality, impactful papers. Many influential ideas in modern machine learning, from residual networks to GANs to transformers, were introduced or gained traction through ICLR. So the research presented here offers a glimpse at the future of AI.
Key Trends from ICLR 2019
Looking at the best papers and other notable works from ICLR 2019, a few broad themes emerge:
- Innovations in neural network architectures, especially Recurrent Neural Networks (RNNs) and Transformers
- Techniques for improving the efficiency of neural networks and reducing their computational/memory footprint
- Theoretical frameworks and analysis tools to better understand neural networks and guide their design
- Applications of deep learning to areas like natural language processing, graph-structured data, reasoning and planning
In this post, we‘ll dive into the paper that won the Best Paper award as well as a few other highlights. These exemplify some of the key research directions in the field.
Best Paper Winner: "Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks"
The winner of the Best Paper award at ICLR 2019 was "Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks" by Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Shen and Courville are from MILA (Montreal Institute for Learning Algorithms) and the Université de Montréal, while Tan and Sordoni are from Microsoft Research.
Motivation and Key Idea
This paper addresses a fundamental challenge in applying recurrent neural networks (RNNs) to natural language: while human language has a hierarchical, tree-like structure, standard RNNs process words purely sequentially without explicitly modeling higher-level constructs like phrases and clauses.
The authors propose a new architecture called Ordered Neurons LSTM (ON-LSTM) that aims to imbue RNNs with an inductive bias towards learning hierarchical structure. The key idea is to impose an order on the neurons (hence "Ordered Neurons"), such that higher-ranking neurons update less frequently and model longer-term, higher-level information while lower-ranking neurons update more often and handle fine-grained details.
Architecture and Algorithms
ON-LSTM extends the standard Long Short-Term Memory (LSTM) architecture with two main innovations:
-
New gating mechanism: In addition to the usual input and forget gates, ON-LSTM introduces a "master" input gate and forget gate. These master gates use a novel activation function called cumulative softmax (cumax) to enforce the ranking of neurons.
-
Cumax activation function: Cumax is defined as the cumulative sum of a softmax. This ensures the gating mechanism is monotonic – as you move from lower to higher-ranking neurons, the master forget gate goes from 0 to 1 while the master input gate goes from 1 to 0. This governs the flow of information between high and low-ranking neurons.
During training, the model learns the optimal ranking of neurons for the task at hand. This equips the RNN with an inductive bias to capture long-term dependencies and recursively compose words into higher-level chunks, without explicit parse trees as supervision.
Results and Impact
The researchers evaluate ON-LSTM on several language tasks:
- Language modeling: ON-LSTM improves perplexity over standard LSTM on the Penn Treebank and WikiText-2 datasets
- Unsupervised parsing: When trained on language modeling, the gate activations in ON-LSTM correlate with syntactic structure and can be used for unsupervised constituency parsing, outperforming prior methods
- Natural Language Inference: ON-LSTM achieves state-of-the-art results on the SNLI dataset, demonstrating its ability to capture semantic relationships
This work demonstrates how imbuing neural networks with structured inductive biases can improve their ability to learn meaningful representations and tackle complex tasks. The imposition of hierarchical structure also points to the benefits of integrating classical ideas from linguistics and symbolic AI with modern deep learning.
While further analysis and extensions are needed, Ordered Neurons is a step towards deep learning architectures that are more explainable, data-efficient, and robust. It opens up avenues for better language understanding, unsupervised structure discovery, and reasoning over longer contexts.
Notable Runner-Up: "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks"
Another highly influential paper from ICLR 2019 was "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" by Jonathan Frankle and Michael Carbin from MIT CSAIL. This paper introduced the provocative idea that within large, dense neural networks, there exist much smaller subnetworks that can be trained in isolation to achieve comparable performance.
The key findings are:
- Dense, randomly-initialized networks contain sparse subnetworks ("winning tickets") that can be trained to achieve similar accuracy as the original network
- These winning tickets can be identified by iteratively pruning the lowest-magnitude weights of a trained network, then rewinding the remaining weights to their initial values
- The initializations of winning tickets are crucial – when randomly reinitialized, they learn slower and perform worse
- Winning tickets are task-specific and have an inductive bias well-suited to the problem – the researchers hypothesize that the structure of the sparse network encodes useful priors
This work has important implications for neural network compression, architecture search, and understanding why overparameterized networks generalize well. It suggests significant redundancy in standard architectures and points to a need for techniques beyond pruning that can discover optimal sparse structures.
Follow-up work by the same authors and others have extended the lottery ticket hypothesis to larger-scale settings, different architectures, and reinforcement learning. There remain open questions, however, around the limits and generality of this phenomenon, how to find winning tickets more efficiently, and what makes their initializations special.
Other Highlights
A few other notable papers from ICLR 2019:
- "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context" – introduced a transformer architecture that can learn long-term dependencies, achieving SOTA on several language tasks
- "Deep Graph Infomax" – proposed a general framework for learning unsupervised representations of graph-structured data
- "Implicit Generation and Modeling with Energy Based Models" – developed more expressive and controllable alternatives to Generative Adversarial Networks (GANs)
Conclusion
ICLR 2019 showcased significant progress in fundamental deep learning methods as well as innovative applications. The best papers exemplify key trends in the field – incorporation of structural priors, techniques for compression and efficiency, unsupervised and self-supervised learning, and unified frameworks that bridge deep learning with classical methods.
While deep learning has driven remarkable advances over the past decade, many open challenges remain around data efficiency, interpretability, robustness, and grounding. As the papers covered here demonstrate, however, the field continues to evolve and push forward the capabilities of machine learning. Sustained progress and convergence of ideas from the vibrant research community, as represented at venues like ICLR, will be crucial for fulfilling the grand vision of AI.
