Training and Deploying State-of-the-Art NLP Models with Hugging Face and Amazon SageMaker

Natural language processing (NLP) has undergone a paradigm shift in recent years with the advent of transformer-based models. These deep learning architectures, first introduced in the landmark 2017 paper "Attention is All You Need", have shattered benchmarks across virtually every NLP task, from classification and sequence labeling to translation and summarization to question answering and text generation.

Transformers work by applying an attention mechanism that learns to focus on the most relevant parts of the input sequence. Stacking multiple attention layers allows transformers to learn rich, nuanced representations of language. When combined with transfer learning, which involves pre-training on large general corpora before fine-tuning on specific tasks, transformers yield unprecedented accuracy, often matching or exceeding human-level performance.

To illustrate the leap in capabilities enabled by transformers, here are some key milestones:

• In 2018, BERT (Bidirectional Encoder Representations from Transformers) achieved state-of-the-art results on 11 NLP tasks, outperforming previous RNNs and CNNs
• In 2019, the T5 (Text-to-Text Transfer Transformer) model attained near human-level performance on the challenging SuperGLUE language understanding benchmark
• In 2020, GPT-3 (Generative Pre-trained Transformer 3) demonstrated remarkable language generation capabilities, able to produce creative fiction, coherent dialogue, and working code from minimal prompts
• In 2021, Wu Dao 2.0, a 1.75 trillion parameter model trained on both Chinese and English, set new records in areas like reading comprehension, text summarization, and commonsense reasoning

The following chart shows the rapid growth in transformer model size over the past few years, culminating in today‘s massive billion and even trillion parameter networks:

While these jumbo-sized models are extremely expensive to train from scratch (often costing millions of dollars in compute), they are surprisingly affordable to fine-tune and deploy thanks to open source platforms like Hugging Face. Founded in 2016, Hugging Face has built a thriving ecosystem for cutting-edge NLP, centered around its model hub which hosts over 10,000 pre-trained transformers.

The Hugging Face model hub has seen exponential growth as researchers and practitioners share their carefully tuned models with the world. It now spans more than 100 languages and includes entries from tech giants like Google, Microsoft, Amazon, Baidu, and Nvidia as well as leading universities and government labs.

Using these pre-trained models, it‘s possible for anyone to achieve state-of-the-art performance on their NLP task of interest with relatively little training data or computing power. Hugging Face‘s Transformers library provides a unified interface for loading and fine-tuning models from all the major deep learning frameworks (PyTorch, TensorFlow, JAX).

Amazon SageMaker provides the ideal platform for scaling up Hugging Face workloads to large datasets and production use cases. With its fully managed infrastructure and broad selection of high-performance instances, SageMaker lets you focus on iterating on models rather than juggling servers.

SageMaker tightly integrates with Hugging Face at every stage of the ML lifecycle:

• Data preparation: The SageMaker Processing API enables running Hugging Face‘s tokenizers and feature extractors on large datasets, with automatic scaling and progress tracking
• Training: The SageMaker HuggingFace estimator abstracts away infrastructure complexities, allowing you to fine-tune models using popular architectures like BERT, RoBERTa, ALBERT, and DistilBERT with just a few lines of code
• Hyperparameter optimization (HPO): SageMaker Automatic Model Tuning makes it easy to optimize model hyperparameters through techniques like random search and Bayesian optimization, often improving accuracy by several percentage points
• Distributed training: SageMaker‘s data parallelism and model parallelism libraries allow you to scale training up to hundreds of GPUs, shortening iteration cycles and enabling larger models
• Experiments: SageMaker Experiments lets you organize and track model training runs, recording the parameters, configurations, and outcomes for each experiment
• Debugging and profiling: SageMaker Debugger provides real-time monitoring of system bottlenecks and common failure modes during training, while SageMaker Model Monitor detects concept drift and anomalies in production
• Deployment: SageMaker Endpoints and SageMaker Serverless Inference make it trivial to deploy your trained models as secure, scalable APIs for real-time or batch predictions
• AutoML: For those new to NLP, SageMaker Autopilot automatically builds, trains and tunes the best Hugging Face model for your dataset with just a few clicks

To see these capabilities in action, let‘s walk through an example of fine-tuning a transformer for sentiment analysis on product reviews. We‘ll use the popular BERT model which has been pre-trained on a massive corpus of unlabeled text (Wikipedia + BookCorpus). By learning to predict randomly masked words in its training data, BERT builds a deep bidirectional understanding of language that can then be transferred to downstream tasks.

Here‘s how we can load a pre-trained BERT model and tokenizer from the Hugging Face hub using the transformers library:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)

The AutoModelForSequenceClassification class adds a single linear layer on top of the base BERT model, which we can fine-tune for our binary (positive/negative) sentiment task.

Next we load our product reviews dataset and apply the BERT tokenizer to prepare the text for input to the model:

import datasets
from transformers import DefaultDataCollator

dataset = datasets.load_dataset("rotten_tomatoes", split="train")
dataset = dataset.map(lambda e: tokenizer(e[‘text‘], truncation=True, padding=‘max_length‘), batched=True)
data_collator = DefaultDataCollator(return_tensors="pt")

With our data ready, we can launch a fine-tuning job on SageMaker using the HuggingFace estimator:

from sagemaker.huggingface import HuggingFace

hyperparameters = {
    "epochs": 1,
    "train_batch_size": 32,
    "model_name": model_name    
}

huggingface_estimator = HuggingFace(
    entry_point="train.py",
    instance_type="ml.p3.2xlarge",
    instance_count=1,
    role=sagemaker.get_execution_role(),
    transformers_version="4.6",
    pytorch_version="1.7",
    py_version="py36",
    hyperparameters=hyperparameters,
)

huggingface_estimator.fit()

After a few minutes of training, our sentiment model is ready to deploy:

predictor = huggingface_estimator.deploy(1, "ml.m5.xlarge")

We can then invoke the SageMaker endpoint to analyze the sentiment of new reviews:

review = "This movie was amazing! The acting was superb and the plot kept me on the edge of my seat."
predictor.predict({"inputs": review})

[{"label":"POSITIVE", "score":0.99}]

And there you have it – with just a small labeled dataset and a few lines of code, we‘ve harnessed the power of transformers to build a highly accurate sentiment model!

Of course, text classification is just one of countless use cases for transformer models. The same approach can be adapted to tackle more advanced challenges like extractive question answering, abstractive summarization, machine translation and even language generation. Researchers have shown how transformer language models can be used as knowledge bases, able to store and retrieve facts to answer open-ended questions. Recent work has even demonstrated transformers‘ ability to handle multi-modal tasks involving images, video, and audio when pre-trained on large volumes of paired multimedia data.

Productionizing these powerful models is not without its challenges, however. Fine-tuned transformers are large and computationally demanding, often requiring specialized hardware like GPUs or TPUs for real-time inference. Quantization, pruning, knowledge distillation and other optimization techniques can help compress models to improve latency and throughput, but may impact accuracy.

Monitoring models for performance regressions and data drift is also critical for maintaining quality in a dynamic production environment. SageMaker Autopilot helps automate model tuning and management, while SageMaker Debugger and Model Monitor provide visibility into runtime issues and prediction quality.

Looking ahead, the NLP community is abuzz with potential new directions for pushing transformer models even further. Larger model sizes in the billions and trillions of parameters are pushing the boundaries of what‘s possible with today‘s compute infrastructure. Techniques like retrieval, few-shot learning, active learning, reinforcement learning, and unsupervised pre-training hold immense promise for boosting accuracy and sample efficiency.

Distributed computing and cloud platforms like SageMaker will be essential for training and deploying these next-generation models at scale. With its rich managed services and broad partner ecosystem, SageMaker is well-positioned to accelerate NLP innovation and help take transformers from the lab to the real world.

As the volume of unstructured text data continues to grow exponentially, transformers and platforms like Hugging Face and SageMaker will undoubtedly play a central role in unlocking its value. We‘re excited to see what new breakthroughs emerge at the intersection of NLP, deep learning and cloud computing. There‘s never been a better time to get started with transformers and experience their power firsthand. Happy coding!

References:

• Attention is All You Need (Vaswani et al, 2017)
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al, 2018)
• Language Models are Unsupervised Multitask Learners (Radford et al, 2019)
• SageMaker Hugging Face