# Top 10 Must\-See Machine Learning Tutorials from NeurIPS 2018 - Canonical: https://33rdsquare.com/top-highlights-tutorials-neurips-2018/ - Published: 2024-09-03 - Author: Jordan Brown - Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/) --- ## Introduction The Conference on Neural Information Processing Systems, better known as NeurIPS (formerly NIPS), is widely regarded as the premier annual gathering for the machine learning community. For over 30 years, NeurIPS has brought together leading researchers to share groundbreaking work that continually redefines and advances the state-of-the-art in ML and AI. Securing a coveted spot on the program is no easy feat. Out of thousands of papers submitted each year, only a small percentage survive the rigorous peer review process. Likewise, tutorial proposals face stiff competition, with an acceptance rate typically in the single digits. The 2018 edition of NeurIPS was the biggest one yet, featuring over 1,000 papers and 30 tutorials spread across 9 parallel tracks. Over 8,500 attendees descended upon Montreal for the week-long event, snatching up every available ticket within minutes of release. This explosion in size and popularity underscores the rapid progress and expanding impact of machine learning research. With so much excitement in the air, it was impossible to catch every important talk or poster. Fortunately, the NeurIPS organizers have made most of the content freely available online. After reviewing hours of recordings, I‘ve curated a list of the top 10 tutorials that every ML practitioner should check out. These sessions showcase major trends, practical techniques, and forward-looking ideas that are shaping the future of our field. Whether you‘re looking to stay on top of the latest research, pick up new skills, or draw inspiration for your own projects, you‘re sure to find these tutorials both enlightening and rewarding. So grab a cup of coffee, dig in, and get ready to elevate your machine learning game! ![NeurIPS 2018 Highlights](https://33rdsquare.com/neurips-2018-highlights.jpg) ## 1. Automatic Machine Learning What if you could automate the entire process of applying machine learning to a dataset or problem? That‘s the ambitious yet enticing proposition behind AutoML, a research area that has gained significant momentum in recent years. In this tutorial, Frank Hutter and Joaquin Vanschoren provided a comprehensive overview of methods for automating the design of ML pipelines. This includes techniques for feature engineering, model architecture search, hyperparameter optimization, and more. By intelligently exploring the vast space of possibilities, AutoML systems aim to discover high-performing models with minimal human intervention. The speakers covered both the theoretical underpinnings of AutoML as well as practical considerations for building such systems. Key approaches include Bayesian optimization, evolutionary algorithms, reinforcement learning, and meta-learning. Armed with these tools, data scientists may soon have AI assistants that can help tackle machine learning tasks from start to finish. While AutoML is not yet a solved problem, this tutorial made a strong case for its transformative potential. I believe we will continue to see rapid progress towards truly "hands-free" machine learning in the years ahead. This will not only boost productivity for experts, but also make powerful ML capabilities accessible to a wider audience. ## 2. Visualizing and Interpreting Deep Learning Models Deep neural networks are notorious for being "black boxes," dazzling us with results while keeping their inner workings opaque and mysterious. In domains like healthcare where human oversight is critical, this lack of transparency can present major barriers to adoption. Fernanda Viegas and Martin Wattenberg, legendary pioneers in data visualization, showed us that it doesn‘t have to be this way. Through a series of compelling examples, they demonstrated how thoughtful visualizations can help peel back the curtain on deep learning models, revealing insights about learned representations, attention mechanisms, and more. Some of my favorite techniques they covered: - Activation maps that highlight which parts of an input image are most salient for a given class - Embedding projections that reveal clusters and semantic relationships in the latent space - Attribution heatmaps that explain individual predictions in terms of input features - Interactive tools for probing the role of each neuron in the network Far from an academic exercise, visual interpretability is key to building safe, fair, and trustworthy AI systems. It can help us identify potential sources of bias, catch errors before they cause harm, and understand model limitations. As we entrust more and more high-stakes decisions to deep learning, visualization must play a central role in keeping a human in the loop. ## 3. Adversarial Robustness – Theory and Practice With great power comes great responsibility. As deep learning permeates every corner of industry and government, the security and integrity of these systems is paramount. Unfortunately, it turns out that many state-of-the-art models are quite brittle – cleverly crafted "adversarial examples" can fool them into making implausible errors. Aleksander Madry and Zico Kolter gave a riveting tutorial on the fast-moving field of adversarial machine learning. In the first half, they laid out a precise mathematical framework for analyzing the robustness of ML models. By formulating adversarial attacks as an optimization problem, one can derive provable guarantees about a model‘s worst-case performance in the presence of bounded perturbations. The second half focused on empirical defenses to make models more resilient to attack. A key idea is adversarial training – incorporating adversarial examples into the training set to build immunity. The speakers shared tips and tricks for getting adversarial training to work in practice, as well as their latest research pushing the envelope on robust architectures. Make no mistake – securing machine learning systems is a daunting challenge that will require ongoing vigilance. However, this tutorial equipped practitioners with valuable tools for measuring and mitigating risks before they materialize. I predict that robustness will soon become a standard evaluation metric alongside accuracy, especially for mission-critical applications. ## 4. Unsupervised Learning – The Next Frontier? To many, unsupervised learning represents the "holy grail" of AI – the ability to discover useful structure and representations from raw, unlabeled data. After all, this is how humans seem to learn, absorbing knowledge about the world through observation and interaction rather than explicit instruction. Alex Graves and Marc‘Aurelio Ranzato gave an exciting glimpse into recent progress in this area. They covered a wide range of approaches, including: - Autoencoders and generative models that learn to compress and synthesize data - Contrastive methods that learn representations by comparing pairs of "similar" and "dissimilar" inputs - Exploration objectives that encourage agents to seek novelty and diversity - Meta-learning algorithms that "learn how to learn" from past experience While supervised learning still reigns supreme for most practical applications, unsupervised pre-training can provide a valuable "head start." By learning generic features and representations from massive unlabeled datasets, we can boost performance and sample efficiency on downstream tasks. This "transfer learning" paradigm has already sparked a revolution in natural language processing, with models like BERT and GPT-3 trained on huge swaths of Internet text. As we look to the future, unsupervised learning will be essential for achieving human-like flexibility and generalization. I believe the most transformative AI breakthroughs may come from figuring out how to learn rich models of the world in a self-supervised way. Judging from the enthusiasm of the speakers and audience, much of the NeurIPS community shares this view. ## 5. Bayesian Inference at Scale In machine learning, there are two major schools of thought – the "frequentist" camp which sees model parameters as fixed constants, and the Bayesian camp which sees them as random variables with prior distributions. Bayesian methods offer a principled way to quantify uncertainty, incorporate domain knowledge, and reason about probabilistic relationships between variables. Historically, Bayesian inference has been computationally intense, limiting its use to small datasets and simple models. In this tutorial, David Dunson showed that it‘s possible to scale Bayesian techniques to big data problems with millions of parameters. The key is to combine clever MCMC sampling schemes with distributed systems and hardware accelerators. Some highlights: - Variational inference algorithms that turn posterior approximation into an optimization problem - Stochastic gradient MCMC that scales to huge datasets - Probabilistic programming languages that simplify Bayesian model specification and inference - Case studies applying Bayesian deep learning to computer vision, NLP, and recommender systems I came away impressed by how far Bayesian methods have come in the age of big data. While they may not be appropriate for every problem, they offer a powerful toolkit for exploiting prior knowledge, expressing uncertainty, and learning from limited labeled data. I expect Bayesian and frequentist approaches to continue to cross-pollinate and converge over time. ## Conclusion I hope this gives you a taste of the intellectual smorgasbord on display at NeurIPS 2018. Of course, I‘ve barely scratched the surface – there were many more excellent tutorials that I didn‘t have space to cover here. I encourage you to peruse the full program and watch the videos for yourself. If I had to summarize the main themes, I‘d say that machine learning is evolving rapidly along several fronts: 1. Automating the model development process 2. Making models more interpretable and reliable 3. Expanding the frontiers of unsupervised learning 4. Scaling Bayesian inference to large-scale problems We are witnessing a steady march from artisanal, hand-crafted models to large-scale, self-supervised systems. Of course, there is still much work to be done to make these approaches practical and robust. But the overall trajectory is clear – machine learning is becoming more powerful, more autonomous, and more ubiquitous. As ML practitioners, it‘s on us to keep up with the latest advances so we can put them into practice responsibly and effectively. I hope these tutorials provide a good starting point for your own learning journey. Here‘s to an exciting future for our field – see you at NeurIPS 2019 and beyond! --- Source: [Top 10 Must\-See Machine Learning Tutorials from NeurIPS 2018](https://33rdsquare.com/top-highlights-tutorials-neurips-2018/)