Transfer Learning for NLP: Fine-Tuning BERT for Text Classification

Transfer learning has revolutionized natural language processing (NLP) in recent years, enabling breakthroughs in a wide range of language understanding tasks. By leveraging the rich linguistic knowledge captured in pre-trained language models like BERT, practitioners can achieve state-of-the-art performance with minimal task-specific architecture and labeled data. In this post, we‘ll explore what makes this approach so powerful and walk through the process of fine-tuning BERT for text classification.

Why Transfer Learning Matters in NLP

Historically, building NLP models that could genuinely understand language was severely limited by the scarcity of labeled data. Consider a task like sentiment analysis, which involves predicting whether a piece of text expresses a positive or negative opinion. The nuances of human language make it very difficult to hand-craft rules that accurately capture sentiment across diverse domains. Machine learning offers a more flexible approach, but training a deep neural network from scratch would require massive amounts of labeled examples covering the full diversity of language.

This is where transfer learning comes in. The key insight is that many NLP tasks, while superficially distinct, rely on a common foundation of linguistic knowledge – an understanding of syntax, semantics, pragmatics, and world knowledge. If this general language understanding can be learned in an unsupervised way from large unlabeled text corpora, then it can be transferred to downstream tasks, dramatically reducing the need for labeled data and task-specific architectures.

Pre-trained language models are the most successful approach to transfer learning in NLP today. By training on massive unsupervised corpora to predict the next word in a sequence (language modeling) or corrupted words in a sequence (masked language modeling), these models build rich representations of language that can be fine-tuned for specific tasks with minimal labeled data.

The Rise of BERT

BERT (Bidirectional Encoder Representations from Transformers) has become the most widely-used pre-trained language model since its introduction by Google researchers in 2018. BERT builds on earlier approaches like word embeddings and sequence-to-sequence models, but introduces several key innovations that have enabled it to achieve state-of-the-art performance on a wide range of NLP benchmarks.

Architecture

At its core, BERT is a stack of transformer blocks, each implementing a multi-head self-attention mechanism. In contrast to recurrent models like LSTMs, transformers process sequences in parallel, attending to the full context at each position. This allows them to capture long-range dependencies more effectively while being far more computationally efficient.

BERT also introduces segment embeddings and positional embeddings to represent the structure of the input. Segment embeddings allow BERT to differentiate between sentences, while positional embeddings encode the relative position of each token. This allows BERT to reason about the order and relationships between parts of the input.

The BERT architecture (Image source: BERT paper)

Pre-Training

BERT is pre-trained on a massive corpus of unlabeled text, including the entire Wikipedia (2.5B words) and BookCorpus (800M words). This pre-training is done using two unsupervised tasks:

1. Masked Language Modeling (MLM): Some percentage of input tokens are masked at random, and the model is trained to predict these masked tokens. This forces the model to learn a bidirectional representation of the input, as opposed to unidirectional language models that can only look at the left context.

2. Next Sentence Prediction (NSP): The model is given pairs of sentences and trained to predict whether the second sentence follows the first in the original text. This encourages the model to learn long-range dependencies and discourse structure.

Through this pre-training process, BERT builds a rich representation of language that captures syntax, semantics, and world knowledge. Crucially, this knowledge is bidirectional – the representation of each word takes into account both its left and right context.

Fine-Tuning

Once pre-trained, BERT can be fine-tuned for a wide variety of downstream tasks with minimal architecture changes. For sequence classification tasks like sentiment analysis, a single linear layer is added on top of the pooled output from BERT, representing the entire input sequence. The model is then trained on a small amount of labeled task-specific data, with the BERT weights either frozen or updated at a very low learning rate.

This fine-tuning process leverages the linguistic knowledge captured in the pre-trained BERT model, allowing even small labeled datasets to yield state-of-the-art performance. The BERT paper demonstrates this on the GLUE benchmark, where BERT achieves the best results on all 9 tasks with just a single additional output layer.

Fine-Tuning BERT for Text Classification

Now let‘s dive into the practical details of fine-tuning BERT for a text classification task. We‘ll use the popular IMDb movie review dataset as an example, where the goal is to predict whether a movie review expresses positive or negative sentiment.

Setup

The first step is to load a pre-trained BERT model and its associated tokenizer. The transformers library from Hugging Face makes this easy:

from transformers import BertTokenizer, BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained(‘bert-base-uncased‘, num_labels=2)
tokenizer = BertTokenizer.from_pretrained(‘bert-base-uncased‘)

Here we‘re using the ‘bert-base-uncased‘ model, which is the smaller of the two models from the original BERT paper (110M parameters vs 340M for ‘bert-large-uncased‘). We specify num_labels=2 since we‘re doing binary classification.

Data Preparation

Next, we need to preprocess our text data into a format that BERT can consume. This involves:

1. Tokenization: Splitting the text into word pieces using BERT‘s wordpiece tokenizer. This tokenizer breaks words into subwords, allowing the model to represent a large vocabulary efficiently.

2. Special tokens: Adding special [CLS] and [SEP] tokens to represent the start of the sequence and the separation between sentences.

3. Padding: Padding or truncating all sequences to a fixed length, since BERT requires fixed-size inputs.

4. Attention mask: Creating an attention mask that tells BERT which tokens are padding and should be ignored.

The transformers library provides utilities to handle this preprocessing:

def encode_example(text, max_seq_length=256):
    tokens = tokenizer.tokenize(text)
    tokens = [‘[CLS]‘] + tokens + [‘[SEP]‘]
    padding = [0] * (max_seq_length - len(tokens))
    attention_mask = [1] * len(tokens) + padding
    input_ids = tokenizer.convert_tokens_to_ids(tokens) + padding
    return {‘input_ids‘: input_ids, ‘attention_mask‘: attention_mask}

train_dataset = [encode_example(text) for text, _ in train_data]

Training

With the data preprocessed, we can now fine-tune BERT. The recommended hyperparameters from the BERT paper are:

• Batch size: 16 or 32
• Learning rate: 2e-5, 3e-5, or 5e-5
• Number of epochs: 3 or 4

We‘ll use the AdamW optimizer, which corrects the weight decay issue in the original Adam, and a linear learning rate schedule with warmup.

from transformers import AdamW, get_linear_schedule_with_warmup

optimizer = AdamW(model.parameters(), lr=2e-5)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps=len(train_dataset) * num_epochs)

for epoch in range(num_epochs):
    for batch in train_dataset:
        outputs = model(**batch, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        scheduler.step()
    evaluate(model, dev_dataset)

A few things to note:

• We‘re fine-tuning all layers of BERT, but they‘re initialized with the pre-trained weights and thus need to be updated carefully – hence the low learning rate and linear warmup.
• The model outputs a loss that we backpropagate through the network. The transformers library handles computing this loss under the hood.
• We evaluate on a held-out development set after each epoch to monitor progress and catch overfitting.

Inference

Once the model is fine-tuned, we can use it to make predictions on new text:

def predict_sentiment(text):
    encoding = encode_example(text)
    outputs = model(**encoding)
    scores = outputs.logits.detach().numpy()[0]
    return int(scores[1] > scores[0])  # 1 if positive else 0

The model outputs a score for each label (positive and negative), and we take the argmax to get the predicted class.

Results

So how well does this simple fine-tuning approach work? On the IMDb dataset, BERT achieves an accuracy of around 93.5%, which is state-of-the-art for the dataset. This is impressive given that the model was not pre-trained on movie review text specifically.

We can also look at some examples to get a sense of what the model has learned:

Review	Predicted Sentiment
This movie was absolutely amazing. The acting was superb and the plot kept me on the edge of my seat. I highly recommend it!	Positive
I had high hopes for this film but it fell flat. The pacing was slow, the characters were underdeveloped, and the ending was predictable. Save your money.	Negative
While the special effects were impressive, the story itself was lacking. It felt like a rehash of older sci-fi tropes without adding anything new. An average popcorn flick at best.	Negative

The model does a good job of picking up on both explicit sentiment words ("amazing", "superb", "recommend" vs "flat", "slow", "underdeveloped") and more implicit sentiment ("kept me on the edge of my seat" vs "predictable", "rehash", "average").

Limitations and Future Directions

While BERT has achieved impressive results, it‘s not a silver bullet. There are several important limitations and open problems:

• Computational expense: BERT is a large model and can be slow and memory-intensive to train and serve, especially for long sequences. Techniques like model compression and knowledge distillation can help, but efficiency remains a challenge.

• Lack of interpretability: Like many deep learning models, BERT is essentially a black box. While we can visualize attention patterns to get some insight into what the model is focusing on, it‘s difficult to explain its predictions in a human-understandable way. This can be problematic for applications where interpretability is important.

• Bias: BERT is trained on web data, which can encode societal biases around gender, race, religion, etc. If not carefully controlled for, these biases can propagate to downstream tasks. Techniques like debiasing embeddings and adversarial training can help mitigate this issue.

• Domain adaptation: While BERT‘s linguistic knowledge transfers remarkably well to many domains, performance can still degrade on highly specialized language (e.g. legal contracts, scientific papers). Continued pre-training on in-domain data can help, but collecting large amounts of unlabeled text for every domain is not always feasible.

Despite these limitations, BERT and its successors (e.g. RoBERTa, XLNet, ALBERT) have become the foundation of modern NLP. Active areas of research include more efficient pre-training and fine-tuning, model compression, multimodal models (e.g. combining text with images/speech), few-shot learning, and cross-lingual models. As these techniques mature, we can expect language models to become even more powerful and widely applicable.

Conclusion

Transfer learning with pre-trained language models like BERT has revolutionized NLP, enabling practitioners to achieve state-of-the-art performance on a wide range of tasks with relatively little labeled data. By leveraging the rich linguistic knowledge captured through self-supervised pre-training, BERT can be quickly fine-tuned for text classification and other language understanding tasks.

While there are important limitations and open research problems, BERT and its successors have become an indispensable tool for anyone working in NLP today. As the field continues to advance at an astonishing pace, staying on top of the latest techniques is essential for both researchers and practitioners.