How Google Used Billion-Parameter Neural Networks to Revolutionize Writing Emails

Writing is at the heart of how we communicate and express ourselves in the digital age. Whether crafting a lengthy report or dashing off a quick reply, we spend an inordinate amount of time and effort transforming thoughts into written language. What if AI could help us write better and faster, taking care of the routine phrases and low-level mechanics so we can focus on high-order thinking and creative expression?

That‘s the promise of AI-assisted writing tools like Gmail‘s Smart Compose, which uses cutting-edge natural language processing to suggest text in real-time as you draft messages. Launched in 2018, Smart Compose was a breakthrough in consumer AI – the first time a large neural language model was deployed at the scale of a billion users.

While tools like autocomplete and spell check have long helped us avoid typos, Smart Compose was far more ambitious: a deep learning system that could meaningfully boost the speed and quality of written communication for everyone. Let‘s take a deep dive into how Google built it and what it means for the future of writing.

The Power of Language Modeling

At its core, Smart Compose is powered by a technique called language modeling – the use of statistical and machine learning algorithms to predict the likelihood of a sequence of words. Language models, which assign probabilities to sequences of text, have been a fundamental component of NLP applications for decades.

The key breakthroughs in recent years have come from applying powerful deep learning models to the task of language modeling. By training multi-layer neural networks on massive text datasets, researchers have created models that can generate remarkably fluent, contextual and open-ended text.

The most prominent example is OpenAI‘s GPT-3, a language model with 175 billion parameters trained on a vast swath of internet text. GPT-3 can engage in open-ended conversations, answer questions, and even generate coherent articles and stories in response to prompts. It‘s the largest language model ever created.

While Smart Compose is not as extensive as GPT-3, it applies the same core approach of using state-of-the-art neural networks for language modeling. The key difference is that rather than open-ended generation, Smart Compose is focused specifically on predicting the next words and phrases a user is likely to write based on context.

The Technical Challenges and Cutting-Edge Solutions

Deploying a large neural language model to assist a billion users in real-time presented unprecedented technical challenges for Google‘s AI team. In a research paper, they outlined some of the key hurdles and how they solved them:

Achieving Real-Time Latency at Scale

To provide a useful writing experience, Smart Compose needed to offer suggestions with virtually no noticeable lag, ideally within just 100 milliseconds. That‘s an extremely high bar for a system processing billions of requests.

Initially, the team experimented with a standard LSTM recurrent neural network architecture, a common approach for sequence modeling tasks. However, even running on accelerated hardware, this model was too computationally intensive to meet the latency target.

They then tried Transformer-based language models, which in recent years have achieved state-of-the-art performance on NLP tasks. Specifically, they used a Transformer Sequence-to-Sequence approach with attention, which could capture long-range dependencies in email text.

While transformers surpassed LSTMs in predictive quality, they were even more computationally demanding. So the team developed a novel hybrid approach: a lightweight linear model for rapid initial word and phrase suggestions, combined with a transformer model for more nuanced predictions as context accumulates through the course of writing.

To make this dual-model architecture performant enough for real-time use, Google took advantage of its custom Tensor Processing Unit (TPU) chips. They slimmed down the transformer model and quantized it to accelerate inference on TPUs. Ultimately, this approach met the speed requirements while still preserving sophisticated predictive capabilities.

Training Effectively on Billions of Examples

Smart Compose predicts what you want to write based on the text of billions of real emails. Training a model on such a vast corpus presents both technical and data pipeline challenges.

On the technical side, model training is hugely computationally intensive and time-consuming at this scale, even on accelerators. To make training practical, Google used a highly parallel distributed training setup across many TPUs. They also employed optimization techniques like mixed precision training and learning rate decay.

In terms of data, processing and shuffling billions of individual emails is a major bottleneck. The team built an efficient streaming data pipeline using Google‘s Flume framework to feed training examples to the models. They also applied filtering to remove low-quality and duplicate data.

Ensuring Fairness, Safety and Privacy

As a system designed to foresee what someone will write, deployed to a billion users, Smart Compose had to be extremely robust in terms of fairness and safety. The model needed to avoid generating biased, hateful, explicit or otherwise inappropriate suggestions.

The primary way Google solved this was by carefully curating the training data. They filtered out emails containing sensitive content and personally identifiable information. They also developed classifiers to detect unsafe text generated by the model and suppress those suggestions.

To preserve privacy, all email data was encrypted end-to-end and only used to train aggregate models. No individual user data was used to customize suggestions. The Smart Compose models can only see text typed in the current draft, not previous emails.

After clearing these technical hurdles, Gmail Smart Compose launched to all users in 2018 and quickly became a popular feature. Internal data shared by Google showed that:

• Smart Compose saved users millions of keystrokes every week
• Suggested text was accepted 12.5% of the time
• Emails written with Smart Compose were on average 3 words longer
• Weekly active users increased 350% in the first 5 months
• Smart Compose sped up email writing by 8.5% on average

Advancing the State of the Art

Since the initial launch, Google has continued to improve Smart Compose‘s capabilities using more advanced natural language understanding models and techniques.

One key area of enhancement has been personalization. Unlike the initial release, which used the same global model for all users, Smart Compose now adapts to each person‘s individual writing style. By learning from your sent messages, it can tailor suggestions to your unique voice and preferences.

Smart Compose also gained the ability to suggest email subjects, using a separate model from the body text predictor. Subjects pose a distinct challenge because there is less preceding context to draw on. To account for this, the subject suggestion model was trained on email metadata like recipients and thread history.

More recently, Google announced an experimental effort to expand Smart Compose beyond Gmail to Google Docs, where it can assist with longer-form content like reports and essays. This poses novel challenges, as documents tend to have richer structure and formatting.

To handle the complexities of document-level suggestions, Google is leveraging even more powerful pretrained language models like BERT that can understand context over longer spans of text. They are also developing models that can suggest formatting, not just text, by learning document structure.

Implications for the Future of Writing

Smart Compose offers a glimpse into a future where AI functions as an ever-present cognitive companion, augmenting our efficiency and eloquence across a wide range of knowledge work. By alleviating some of the cognitive load of writing, language models could help us communicate faster and more fluently.

This technology holds particular promise for non-native speakers and those with language or learning disorders. Language models could help level the playing field by providing real-time grammatical and stylistic assistance. They may also find use in educational settings to provide feedback and coaching for novice writers.

At the same time, the rise of AI-assisted writing surfaces important considerations around authenticity, originality and skill development. Will predictive text make our writing more bland and homogeneous over time? Will it erode our innate language faculties if we overly rely on AI suggestions to guide our writing?

There are also critical ethical questions to grapple with, such as bias and fairness. Language models trained on internet data have been shown to reflect discriminatory stereotypes around gender, race and other characteristics. As these models increasingly mediate how we write and communicate, we must be vigilant about identifying and mitigating unfair bias.

Privacy and security will be paramount as well, especially as models become more personalized. Users need assurance that their data is being protected and not misused. The principle of federated learning, where data is processed on-device rather than centralized servers, could help preserve privacy.

Ultimately, realizing the full potential of AI-assisted writing in a responsible manner will be an ongoing challenge. It will require continued technical innovation in natural language processing coupled with proactive, interdisciplinary collaboration to address social impacts.

If we can strike the right balance, AI could amplify human potential and revolutionize the way we transform thoughts into language. Smart email suggestions are just the first step on this journey – in the coming years, AI-powered writing aids could become as ubiquitous and impactful as spell checkers and thesauruses are today. Google‘s pioneering work on Smart Compose lights the way forward.