A Deep Dive into BERT Tokenization: An NLP Expert‘s Guide

Tokenization is a fundamental task in natural language processing (NLP) that involves splitting text into smaller units called tokens. These tokens are what we actually pass into our models for training and inference. The quality of tokenization is critical, as it determines what information is visible to the model and how it generalizes to new data.

While tokenization has always been important, it has taken on new significance with the rise of transformer-based language models like BERT (Bidirectional Encoder Representations from Transformers). These models are pre-trained on massive amounts of unlabeled text data using self-supervised objectives like masked language modeling. This allows them to learn general-purpose language representations that can then be fine-tuned for specific tasks.

Because BERT operates on tokens rather than raw text, its performance is intimately tied to the tokenization scheme used during pre-training and fine-tuning. In this guide, we‘ll take an in-depth look at how BERT tokenization works, best practices for effective tokenization, and important considerations when using BERT for downstream tasks.

The Need for Subword Tokenization

Traditionally, NLP models have used word-level tokenization, where each token represents a complete word. However, this approach has several drawbacks:

1. Vocabulary size: With word-level tokenization, the vocabulary size grows linearly with the size of the training corpus. For large datasets, this can result in vocabularies with millions of words, which is computationally infeasible.

2. Out-of-vocabulary (OOV) words: No matter how large the vocabulary, there will always be rare words that are not included. Word-level models typically handle these OOV words by mapping them to a special "unknown" token, which loses all semantic information.

3. Inflected forms: In morphologically rich languages, a single lemma can have many inflected forms (e.g. "run", "runs", "running"). Treating these as separate words ignores their inherent similarity.

To overcome these issues, BERT uses a subword tokenization algorithm called WordPiece. Originally developed for machine translation, WordPiece is a data-driven approach that strikes a balance between capturing meaning and limiting vocabulary size.

How WordPiece Tokenization Works

WordPiece is a greedy tokenization algorithm that relies on a pre-defined vocabulary of subword units. This vocabulary is learned during pre-training using an approach similar to byte pair encoding (BPE).

The key idea is to start with a small set of characters and iteratively merge the most frequent pairs of symbols to create new vocabulary items. This process continues until a desired vocabulary size is reached (for BERT, this is typically 30,000 tokens).

Here are the general steps for WordPiece tokenization:

1. Initialization: Start with a basic character vocabulary (e.g. lower-case English letters plus a few punctuation symbols).

2. Merge frequent pairs: Identify the most frequent pair of symbols in the training corpus and merge them into a new symbol, adding it to the vocabulary.

3. Repeat: Iteratively merge the next most frequent pair until the target vocabulary size is reached.

4. Tokenize: To tokenize new text, greedily match the longest possible subword units from the learned vocabulary.

For example, suppose we have the following vocabulary:

[‘a‘, ‘b‘, ‘c‘, ‘d‘, ‘e‘, ‘f‘, ‘g‘, ‘h‘, ‘##a‘, ‘##b‘, ‘##c‘, ‘##d‘, ‘##e‘, ‘##f‘, ‘##g‘, ‘##h‘]

To tokenize the word "abcdefgh", the WordPiece algorithm would produce:

[‘a‘, ‘##b‘, ‘##c‘, ‘##d‘, ‘##e‘, ‘##f‘, ‘##g‘, ‘##h‘]

The ‘##‘ symbols indicate that a token is a subword continuation, rather than the start of a new word.

If a word cannot be exactly represented by subword units, WordPiece falls back to individual characters. For example, "xyz" would be tokenized as:

[‘x‘, ‘y‘, ‘z‘]

Special Tokens

In addition to the subword tokens, BERT reserves a few special tokens:

• [CLS]: Prepended to every input sequence. The final hidden state of this token is used as the aggregated sequence representation for classification tasks.
• [SEP]: Used to separate different parts of the input (e.g. sentence pairs).
• [MASK]: Used when pre-training with the masked language modeling objective.
• [PAD]: Used to pad sequences to a fixed length in a batch.

The tokenizer automatically adds these special tokens as needed for each task.

Using the BERT Tokenizer

The easiest way to use the BERT tokenizer is via the 🤗 Transformers library from HuggingFace. This library provides pre-trained tokenizers for all of the popular transformer models.

Here‘s a simple example of using the BERT tokenizer in Python:

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained(‘bert-base-uncased‘)

text = "I love my samoyed dog Lupo!"

tokens = tokenizer.tokenize(text)
print(tokens) 
# [‘i‘, ‘love‘, ‘my‘, ‘samo‘, ‘##yed‘, ‘dog‘, ‘lu‘, ‘##po‘, ‘!‘]

input_ids = tokenizer.encode(text)
print(input_ids)  
# [101, 1045, 2293, 2026, 28440, 25509, 3899, 7322, 24012, 999, 102]

The tokenize method splits the text into WordPiece tokens, while encode adds the special tokens and converts everything to integer IDs.

Tokenization Best Practices

Getting tokenization right is crucial for good model performance. Here are some best practices to keep in mind:

1. Use the same tokenizer for pre-training and fine-tuning. The vocabulary and special tokens need to match between these stages.

2. Be aware of max sequence length. BERT has a hard limit on the number of tokens per sequence (usually 512). Longer sequences will be truncated.

3. Monitor OOV rates. If a large fraction of your tokens are OOV, it may indicate a domain mismatch with the pre-training data. Consider using a domain-specific tokenizer.

4. Handle multiple languages. The default BERT tokenizer is trained on English data. For other languages, use a multilingual model or train a custom tokenizer.

5. Visualize your tokens. For debugging and analysis, it‘s often helpful to look at the actual tokenized representation of your text. The convert_ids_to_tokens method is useful for this.

Tokenization and Model Performance

The quality of tokenization directly impacts downstream model performance. If the tokenizer fails to capture important semantic units or introduces too many OOV tokens, the model will struggle to learn meaningful representations.

Some common issues to watch out for include:

• Over-splitting of rare words: If a word is split into too many subword units, the model may fail to capture its meaning. This is especially problematic for domain-specific terminology.

• Under-splitting of common words: Conversely, if frequent words are not split into enough subwords, the model will treat different inflected forms as entirely separate, ignoring their shared semantics.

• Inconsistent handling of special characters: Characters like hyphens, apostrophes, and quotes can be tricky to handle consistently. Inconsistencies here can result in spurious duplicate tokens.

If you suspect a tokenization issue is hurting your model‘s performance, some things to try include:

1. Visualizing the tokenized inputs to spot any obvious irregularities
2. Comparing the vocabulary overlap between your data and the pre-training corpus
3. Experimenting with different vocabulary sizes and pre-training data
4. Fine-tuning the tokenizer on domain-specific data

Tokenization for Other Transformer Models

While we‘ve focused on BERT in this guide, the same general principles apply to other transformer models. However, there are a few key differences to be aware of.

For example, RoBERTa uses a different pre-training approach that results in a tokenizer without the [CLS] and [SEP] tokens. ALBERT uses a special sentence order prediction (SOP) objective that requires a different input format.

When working with a new transformer model, always consult the documentation to understand its specific tokenization scheme and input format.

The Future of Tokenization

BERT and its contemporaries have achieved impressive results across a wide range of NLP tasks, but there‘s still room for improvement. One active area of research is tokenization for multilingual models.

Current approaches like multilingual BERT (mBERT) simply train on a mixture of languages with a shared vocabulary. However, this doesn‘t optimally handle the unique characteristics of each language. Techniques like language-specific subword regularization have shown promise for improving multilingual tokenization.

Other work has explored alternatives to fixed vocabularies, such as hash-based tokenization and subword regularization. These approaches can potentially handle the open vocabulary problem more gracefully.

As transformer models continue to evolve, tokenization practices will need to evolve with them. It‘s an exciting time for NLP, and we can expect to see many more innovations in this space in the coming years.

Conclusion

In this guide, we‘ve taken a deep dive into BERT tokenization, covering:

• The motivation for subword tokenization
• How the WordPiece algorithm works
• Using the HuggingFace Tokenizers library
• Best practices for effective tokenization
• The relationship between tokenization and model performance
• Tokenization for other transformer models
• The future of tokenization research

Effective tokenization is a critical component of building state-of-the-art NLP systems with BERT and other transformer models. By understanding the inner workings of tokenization and following best practices, you can ensure that your models are learning from the best possible representations of your text data.

Of course, tokenization is just one piece of the puzzle. Successful NLP also requires careful data preprocessing, model architecture design, hyperparameter tuning, and more. However, getting tokenization right is a key foundation upon which everything else is built.

As you apply BERT and other transformer models to your own NLP tasks, keep the lessons of this guide in mind. And as always, happy tokenizing!