The State of AI in 2019: Breakthroughs and Trends

The artificial intelligence revolution is well underway, with both the research community and industry making incredible strides in recent years. What started as a niche discipline has become increasingly ubiquitous, with AI systems starting to impact everything from healthcare and transportation to education and entertainment.

For AI practitioners, simply keeping up with the rapid pace of innovation can feel like a full-time job in itself. Groundbreaking research papers, powerful open source tools, and impressive real-world applications emerge on a seemingly daily basis.

As we enter the second half of 2019, let‘s take a step back and reflect on some of the most impactful developments from the past year. We‘ll also peer ahead at emerging trends that could come to fruition over the next 12 months. While impossible to capture every important advance in a single article, this piece will aim to highlight a sampling of key innovations across major areas like natural language processing, computer vision, automated machine learning, and more.

Whether you‘re an experienced data scientist, an engineer interested in applying ML to your work, or a business leader seeking opportunities to leverage AI, this post will help orient you to the state of the field in 2019. The revolution shows no signs of slowing down – in order to stay relevant, we must all make an effort to stay informed. Let‘s dive in!

2018: A Landmark Year for Language, Vision, and Learning

Natural Language Processing

Machines that can comprehend and communicate in natural language have long been a "holy grail" of artificial intelligence. While still an unsolved challenge, 2018 yielded multiple major breakthroughs that brought us closer than ever before.

Leading the charge were a series of powerful transfer learning approaches, which leverage knowledge gained from pretraining on large general datasets to rapidly adapt to new tasks:

• ULMFiT (Universal Language Model Fine-Tuning), proposed by fast.ai‘s Jeremy Howard and Sebastian Ruder, kicked things off by showing strong results on text classification
• ELMo (Embeddings from Language Models) from AllenNLP captured rich context-dependent word embeddings from bidirectional language models
• Most famously, BERT (Bidirectional Encoder Representations from Transformers) from Google achieved state-of-the-art results on 11 NLP tasks!

These methods demonstrated the incredible power of unsupervised pretraining to learn general language representations that can significantly boost performance on downstream applications. Off-the-shelf code and models have greatly democratized access to cutting-edge NLP. Increasingly, one can attain excellent results without massive hand-labeled datasets or months of training.

Other 2018 highlights included impressive progress on dialog systems, as exemplified by Google Duplex‘s ability to carry out natural conversations to complete tasks like appointment scheduling. We also saw the release of useful tools like Facebook‘s PyText framework and fast.ai‘s "NLP for Coders" course.

Computer Vision

Machines that can perceive and understand visual information have progressed rapidly thanks to the deep learning boom. While groundbreaking vision architectures have become less frequent, 2018 saw continued refinement of existing techniques to yield remarkable results.

Realistic image generation via Generative Adversarial Networks (GANs) was a notable theme. The introduction of BigGANs from DeepMind produced synthetic images that can fool even the most discerning human eye in many cases. We‘re not far from a world in which fake visual content is completely indistinguishable from reality.

Computational efficiency was another key focus, with fast.ai demonstrating how to train an accurate ImageNet classifier in just 18 minutes using a cluster of Amazon Web Services GPU instances costing only $40. While a publicity stunt in some regards, this showcased the rapidly diminishing computational barriers to vision research.

Video processing also saw exciting developments, headlined by NVIDIA‘s vid2vid framework for high resolution video-to-video translation. One can imagine myriad potential applications, from converting simple sketches into fluid animations, to replacing actors‘ faces and bodies in post-production.

Tools, Frameworks and Automated ML

The relentless pace of algorithmic advances has been complemented by a vibrant ecosystem of open source tools and libraries. TensorFlow and PyTorch have emerged as the dominant deep learning platforms, with strong industry adoption and thriving communities.

PyTorch 1.0 launched in late 2018, bringing performance and stability enhancements that have cemented its preferred status among many researchers. The framework powers multiple Facebook products and services at immense scale.

The promising paradigm of automated machine learning (AutoML) gained steam, with the introduction of Auto-Keras for deep learning and continued development of platforms like H20.ai and Google Cloud AutoML. The dream of artificial intelligence to help build artificial intelligence is starting to become reality. While no panacea, AutoML tools can boost efficiency by automating tasks like neural architecture search, hyperparameter tuning, and model selection.

Separately, the release of TensorFlow.js opened up intriguing possibilities for ML in the browser. Pre-trained models can now be deployed and re-trained in client-side JavaScript applications.

Reinforcement Learning

Teaching agents to learn and make decisions through interaction, rather than supervised datasets, remains a stubborn challenge. However, 2018 did see some algorithmic and educational strides.

Google released the Dopamine framework to facilitate flexible, reproducible RL research. It aims to fill a similar niche to OpenAI‘s Gym and DeepMind‘s lab environments.

On the educational front, OpenAI open sourced their "Spinning Up in Deep RL" set of resources, providing a well-documented package of theory, code examples, and learning exercises. This helps satisfy a growing appetite to demystify this sometimes esoteric field.

While not yet yielding any major real-world applications, RL techniques continue to rack up achievements in games and simulations. The coming years will likely see the jump from virtual to physical domains as approaches for sample-efficient learning in complex environments continue to improve.

The Road Ahead: Trends to Watch in 2019 and Beyond

Doing More with Less Labelled Data

Data is the fuel of modern AI, but it can be prohibitively expensive and time-consuming to collect and annotate large datasets for supervised learning. Emerging paradigms like semi-supervised learning, meta-learning, self-supervised learning, weakly-supervised learning, and active learning hint at a future in which high-quality models can be built with fewer manual labels.

2019 has already seen promising results for semi-supervised learning in computer vision, with Google‘s S4L approach achieving 92% of the performance of purely supervised learning while using only 1% as much labeled data! Expect more researchers and practitioners to embrace these data-efficient methods going forward.

ML for Tabular Business Data

Deep learning has largely focused on unstructured data like images, audio, and text. However, structured tabular data still powers key applications across industries like finance, healthcare, retail, and advertising. Tools and best practices for neural networks on tabular data have lagged behind those for other domains.

Fortunately, this is starting to change. Several AutoML-style tools have launched that specifically target supervised learning on tables, aiming to outperform traditional techniques like random forests and gradient boosting. As business increasingly look to adopt ML, there will likely be growing effort devoted to pushing the state-of-the-art on tabular tasks.

Productionizing and Scaling ML

As ML matures and moves from research to production, an entirely new set of challenges around deployment, monitoring, and maintenance emerge. An ecosystem of MLOps tools and managed platforms is blossoming to help data science teams collaborate with engineering and IT to integrate models into products.

From startups to cloud giants, expect major growth in solutions to help instrument, debug, version, scale, and secure models running in the real-world. ML is increasingly not just a model, but an integral component of complex software systems.

Discovering Neural Network Architectures

While our conceptual understanding of deep learning has advanced, much of the empirical progress has been driven by large-scale trial and error experiments to discover architectures well-suited for particular problems. Neural architecture search (NAS) aims to automate the discovery of optimal designs.

Approaches for efficient architecture search, such as differentiable NAS and progressive NAS, are gaining traction. As compute becomes cheaper and AutoML techniques mature, the ability to automatically construct compact, fast models for a wide variety of applications will be very attractive. Imagine generating highly-optimized neural nets for your domain with the push of a button!

Explainable and Fair AI

With AI systems starting to make high-stakes decisions that affect people‘s lives, issues of transparency, accountability, and fairness have come to the forefront. When an algorithm denies you a loan, or determines your health insurance premium, or makes a mistake that causes harm, "the model said so" is not an acceptable answer.

Governments are starting to enact regulations like GDPR which give consumers insight into how their data is used and the right to opt out of automated decision making. The research community is devoting increasing effort to techniques for interpreting the predictions of black-box models and detecting potential bias.

2019 will likely see ethics becoming a mainstream component of AI development across companies and universities. Expect to see new job titles focused exclusively on responsible AI, more public ethical principles from tech giants, and intensifying collaboration between AI and policy experts. We must proactively address these critical issues in order to sustainably harness the immense benefits of artificial intelligence for good.

Conclusion

2018 was a year of rapid research advances and widening industry adoption of AI and ML technologies. 2019 is shaping up to continue the incredible momentum, while expanding the focus on making systems more accessible, scalable, and accountable. It‘s never been a more exciting time to be working in this space.

However, it‘s also never been more challenging to stay current amidst the deluge of new papers, tools, and applications. The key is to identify the developments that intersect with your domain and to go deep in understanding how they might accelerate your projects. Don‘t get overwhelmed – jump in and start experimenting!

No one can predict exactly what breakthroughs will emerge over the next year, but hopefully this article has highlighted some of the areas ripest for innovation. One thing is for sure – the revolution is still in its early innings. Get ready for a thrilling ride ahead!