The Top Papers You Need to Read from ICML 2018

Introduction

2018 was an exciting year for machine learning research, and there was no better place to learn about the latest breakthroughs than the International Conference on Machine Learning (ICML). ICML is one of the biggest and most prestigious annual gatherings focused on ML and AI. Thousands of leading scientists and practitioners from academia and industry convene each year to present cutting-edge research and exchange ideas.

The 35th edition of ICML took place in July 2018 in Stockholm, Sweden. Out of the hundreds of papers submitted and presented at the conference, a handful stood out from the pack and were recognized with Best Paper Awards. In this post, we‘ll take a deep dive into the winners and some of the other most noteworthy papers from ICML 2018 that every data scientist should know about. Get ready to geek out on some state-of-the-art machine learning!

ICML 2018 at a Glance

Before we jump into the papers themselves, let‘s set the stage with an overview of ICML 2018. The 6-day event featured:

• Over 600 papers presented (accepted from 2500+ submissions)
• 9 tutorials
• 35 workshops
• 3 competitions
• Invited talks from leading researchers like Christopher Bishop and Yee Whye Teh

Topics spanned the full gamut of machine learning, from the theoretical foundations to applications in areas like healthcare, robotics, computer vision, and more. There was also a focus on important issues facing the field like fairness, transparency, privacy, and security.

No matter your specific area of interest within ML, odds are there were multiple relevant papers and sessions. For those not fortunate enough to attend in person, the conference proceedings are available online. But with so much ground to cover, knowing where to start can be daunting. That‘s where the Best Paper Awards come in – they recognize work deemed by the conference organizers to be especially significant, novel, and potentially impactful.

Best Paper Award: "Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples"

Adversarial examples are inputs that have been intentionally designed to fool machine learning models into making wrong predictions. A classic example is an image that looks like a panda to human eyes, but is classified as a gibbon by an image recognition system. These malicious inputs pose a serious threat, especially in security-sensitive applications.

As awareness of adversarial attacks has grown, so have efforts to defend against them. One popular approach is gradient masking, which aims to hide information about the model‘s gradients (partial derivatives) from a potential attacker. The basic idea is that if the attacker can‘t compute reliable gradients, then they can‘t use those gradients to craft adversarial examples.

However, in their award-winning paper, a team of researchers demonstrated that obfuscated gradients give a false sense of security. Through theoretical arguments and experiments, they showed how an attacker can circumvent gradient masking to generate adversarial examples that fool the model.

The team identified three main ways that gradient masking can fall short:

1. Shattered gradients – the defense is built on non-differentiable operations that "shatter" the gradient signal
2. Stochastic gradients – the defense injects noise or randomness into the gradients
3. Vanishing/exploding gradients – the defense relies on gradients being very close to 0 or very large in magnitude

For each of these failure modes, the researchers devised techniques that could still generate successful adversarial examples. Through case studies on models defended using these techniques, they demonstrated the techniques‘ effectiveness in circumventing the defenses.

This research serves as an important cautionary tale about the limitations of gradient masking as an adversarial defense. It shows that many proposed defenses are still vulnerable to carefully crafted attacks, even if those attacks are harder to pull off. Going forward, more robust and well-rounded defense strategies will be needed to address the threat of adversarial examples.

Best Paper Award: "Delayed Impact of Fair Machine Learning"

As machine learning is deployed in an ever-expanding range of high-stakes domains, there are growing concerns about fairness and potential negative consequences for certain groups. We‘ve seen multiple real-world examples of bias creeping into ML systems for applications like hiring, lending, criminal sentencing, and more.

Most existing approaches to fairness in ML aim to impose constraints to equalize some metric across groups at the point when the model makes a decision. For example, the percentage of loan applicants approved might be constrained to be equal for different racial groups.

However, the winning paper "Delayed Impact of Fair Machine Learning" questions whether this is sufficient in the long run. Through mathematical modeling and experiments, the authors show how fairness interventions can have unexpected long-term consequences as populations evolve over time. Decisions made now can influence the underlying characteristics of different groups in the future.

As a hypothetical example, suppose an ML system for college admissions imposes a fairness constraint to equalize acceptance rates across income groups. The constraint helps boost lower-income students in the short term. But over time, it could lead to changes in the applicant pool – such as more lower-income students being motivated to apply – that actually exacerbate disparities.

The paper presents a one-step feedback model of decision-making that surfaces these potential long-term impacts. The model can be used to simulate how group characteristics and decision outcomes shift over time as a result of fairness constraints.

Through experiments on real-world datasets, the researchers demonstrate that dynamics like the hypothetical above can indeed play out. Constraints that look effective in the short term can lead to widening gaps after multiple model training cycles.

This work calls for a rethinking of the standard ML fairness paradigm. Rather than narrowly focus on a point-in-time snapshot, fairness considerations need to grapple with the ripple effects of decisions over time. It‘s a complex issue that requires drawing on ideas from causal inference, feedback control, and more.

Other Notable Papers

Beyond the two best paper winners, ICML 2018 featured many other groundbreaking pieces of research. Here are a few more highlights:

• "The Mechanics of n-Player Differentiable Games" – This paper, which won an honorable mention award, studies machine learning in the context of multi-agent games where each agent aims to maximize their own reward. It provides insights into the dynamics of finding equilibria in these settings.

• "Near Optimal Frequent Directions for Sketching Dense and Sparse Matrices" – Sketching is a technique for compressing large matrices while preserving key information. This paper (another honorable mention winner) gives a new approach to sketching based on the frequent directions algorithm.

• "Fairness Without Demographics in Repeated Loss Minimization" – Continuing the theme of ML fairness, this paper proposes a new approach that avoids the need to explicitly specify protected demographic groups. Instead, it aims to ensure that losses are evenly distributed across all groups.

• "Sever: A Robust Meta-Algorithm for Stochastic Optimization" – Stochastic optimization underpins many ML approaches, but is sensitive to noise and outliers in the data. This paper introduces a new meta-algorithm called Sever that can adapt to noise and contamination to give more robust performance.

Perspectives and Takeaways

Reflecting on the top papers from ICML 2018, a few key themes emerge:

• Robustness – Multiple papers explore ways to make ML systems more robust, whether to adversarial attacks, noisy data, or other perturbations. There‘s a growing recognition that brittleness is a major weakness of many models.
• Fairness – ML fairness continues to be a major topic of research interest. We‘re seeing new mathematical formulations of fairness that go beyond simple parity constraints. There‘s also more consideration of the long-term impacts of deploying "fair" models.
• Theoretical underpinnings – Sprinkled among the more application-oriented papers are key theoretical advances. Work like the n-player games paper helps build the conceptual foundations upon which sound machine learning practice rests.

Of course, this only scratches the surface of all the research presented at ICML 2018. But hopefully it gives a flavor of some of the most exciting developments. What stands out to me is how the conference showcases both the immense power and the limitations of current machine learning techniques. Yes, we can build amazingly capable models, but they can be brittle, biased, and hard to interpret. Grappling with these limitations is a major thrust of current research.

For data scientists and ML practitioners, keeping tabs on research trends is crucial for staying at the top of your game. While it can be daunting to wade through hundreds of dense academic papers, focusing on the most celebrated work is a good starting point. The ICML best papers offer a window into what problems are top-of-mind for leading researchers and what techniques are at the cutting edge.

But we shouldn‘t just treat these papers as intellectual novelties – the insights they provide can and should inform real-world practice. For example, the research on adversarial attacks and fairness has major implications for building robust and responsible ML systems. Taking these issues seriously is key for instilling trust in ML and ensuring it has a positive societal impact.

Conclusion

ICML is a marquee event for the machine learning community – a place where the best and brightest minds come together to share their latest discoveries. 2018‘s edition did not disappoint, with an array of truly impressive and thought-provoking papers.

The best paper winners on adversarial examples and delayed impacts of fair ML exemplify the kind of research that can really push the field forward. They identify key shortcomings of current approaches and propose novel techniques for addressing them. At the same time, they raise important new questions to grapple with.

For data scientists, engaging with this kind of cutting-edge research, whether from ICML or other top conferences like NeurIPS and ICLR, is invaluable. It can deepen your knowledge, spark new ideas, and ultimately make you better at your craft. So block off some time, brew a pot of coffee, and dig into those proceedings – your models will thank you!