A Deep Dive into Hugging Face‘s Tokenizers Library for NLP

Introduction

In the field of Natural Language Processing (NLP), tokenization is a fundamental pre-processing step that involves breaking down text into smaller units called tokens. These tokens form the building blocks for many downstream NLP tasks like language modeling, text classification, named entity recognition, machine translation, and more.

Tokenization is required because deep learning NLP models cannot directly work with raw text. The text needs to be converted into a numerical representation first, which is where tokenization comes in. By splitting text into tokens, we can map each token to an integer ID and construct vectorized representations that deep learning models can make sense of.

While tokenization sounds simple conceptually, there are many different algorithms and approaches for converting text to tokens. In recent years, more sophisticated tokenization schemes have emerged, such as WordPiece, SentencePiece, and Byte-Pair Encoding (BPE). These subword tokenization methods can handle large vocabularies and out-of-vocabulary words gracefully.

The Hugging Face Ecosystem

Hugging Face is a popular NLP company that has open-sourced several widely used libraries for working with Transformer-based models. Their flagship offering is the Transformers library, which provides a unified API for accessing pretrained models like BERT, GPT, RoBERTa, XLNet, and more. Hugging Face also maintains the Datasets library for easily loading and processing datasets and the Accelerate library for simplifying distributed training.

Alongside these libraries, Hugging Face also develops the Tokenizers library. As the name suggests, Tokenizers focuses solely on providing state-of-the-art tokenization utilities for NLP. It leverages the Rust programming language under the hood to deliver extremely fast tokenization pipelines suitable for processing large volumes of text data efficiently.

What‘s New in Tokenizers v0.8.0?

In March 2023, Hugging Face released version 0.8.0 of the Tokenizers library. This update brings several exciting new features and improvements, such as:

• The ability to encode pre-tokenized sequences of text in addition to raw strings.
• 5-10x faster training of custom tokenizer models.
• Ease of saving trained tokenizer models in JSON format using a single line of code.
• Numerous bug fixes and other under-the-hood enhancements.

For the full release notes, check out the official changelog. In the rest of this post, we‘ll focus on demonstrating the capabilities of the BertWordPieceTokenizer, one of the key tokenizers included in the library.

Getting Started with the BertWordPieceTokenizer

The BertWordPieceTokenizer is the default tokenizer used for the BERT model. BERT stands for Bidirectional Encoder Representations from Transformers and is one of the most impactful NLP models in recent history. Developed by Google, BERT uses a subword tokenization scheme called WordPiece.

WordPiece constructs a vocabulary of subword units by iteratively merging frequent character sequences. This allows the vocabulary to stay relatively small while ensuring each word can be represented by a sequence of tokens in the vocabulary. Rare words are broken down into subwords while frequent words are kept as single tokens.

Let‘s see how to use the BertWordPieceTokenizer in Python. First, make sure you have the tokenizers library installed:

!pip install tokenizers

Next, we import the BertWordPieceTokenizer class:

from tokenizers import BertWordPieceTokenizer

We need to load a pre-trained WordPiece vocabulary to initialize the tokenizer. Hugging Face hosts several common vocabularies for different models. For this example, we‘ll use the vocabulary from the bert-base-uncased model:

vocab_file = "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt"
tokenizer = BertWordPieceTokenizer(vocab_file)

Now we‘re ready to tokenize some text! Let‘s define a sample sentence:

sentence = "Natural Language Processing is a fascinating field spanning computer science and linguistics."

To tokenize the sentence, we simply pass it to the tokenizer‘s encode() method:

output = tokenizer.encode(sentence)

The output is an Encoding object containing the tokenized representation of the sentence. We can inspect the tokens, token IDs, and character offsets:

print(output.tokens)
print(output.ids)
print(output.offsets)

Here‘s what the output looks like:

[‘[CLS]‘, ‘natural‘, ‘language‘, ‘processing‘, ‘is‘, ‘a‘, ‘fascinating‘, ‘field‘, ‘spanning‘, ‘computer‘, ‘science‘, ‘and‘, ‘linguistics‘, ‘.‘, ‘[SEP]‘] [101, 2984, 3032, 7327, 2022, 1010, 7211, 2214, 18857, 3385, 2480, 1998, 10090, 1011, 102] [(0, 0), (0, 7), (8, 16), (17, 27), (28, 30), (31, 32), (33, 44), (45, 50), (51, 59), (60, 68), (69, 76), (77, 80), (81, 92), (92, 93), (0, 0)]

The [CLS] and [SEP] tokens are special tokens added by the BERT tokenizer to denote the start and end of the sentence. The token IDs are the integer representations of each token in the vocabulary. The offsets indicate the character positions in the original sentence that each token corresponds to.

Saving and Loading Tokenizer Models

Training a custom tokenizer on a new text corpus can be computationally expensive, especially for large datasets. Fortunately, the Tokenizers library allows you to save trained tokenizer models to disk and load them later. This is particularly handy if you need to reuse the same tokenizer across multiple scripts or projects.

To save a tokenizer, use the save() method and specify a file path with a .json extension:

tokenizer.save("bert_tokenizer.json")

To load a saved tokenizer, use the from_file() method of the respective tokenizer class:

loaded_tokenizer = BertWordPieceTokenizer.from_file("bert_tokenizer.json")

Make sure to use the same tokenizer class that was used to create the saved file, otherwise you may encounter serialization errors.

Encoding Pre-Tokenized Text

A convenient feature of the Tokenizers library is the ability to encode text that has already been pre-tokenized. This is useful if your text data has undergone some form of tokenization beforehand, but you still need to apply a specific tokenizer on top of it.

To encode pre-tokenized text, pass a list of tokens instead of a string to the encode() method:

pre_tokenized_text = ["Natural", "Language", "Processing", "is", "a", "fascinating", "field", "spanning", "computer", "science", "and", "linguistics."] output = tokenizer.encode(pre_tokenized_text)

The tokenizer will treat each element of the list as a separate token and apply its tokenization algorithm accordingly. The resulting Encoding object will contain the WordPiece tokens obtained from the pre-tokenized input.

Benchmarking Tokenization Speed

To showcase the efficiency of the Tokenizers library, let‘s measure how long it takes to tokenize a large text corpus. We‘ll use the WikiText-103 dataset, which contains over 100 million tokens from Wikipedia articles.

First, download and extract the dataset:

!wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-raw-v1.zip
!unzip wikitext-103-raw-v1.zip

Next, load the training set file and count the number of lines:

with open("wikitext-103-raw/wiki.train.raw", "r", encoding="utf-8") as f:
train_lines = f.readlines()

print(f"Number of lines in training set: {len(train_lines)}")

To tokenize the entire training set efficiently, we can use the encode_batch() method which accepts a list of sentences and processes them in parallel:

%%time
output = tokenizer.encode_batch(train_lines)

On a Google Colab notebook with a standard GPU, tokenizing the 1.8 million lines in the WikiText-103 training set takes just under 3 minutes. This demonstrates the impressive speed of the Tokenizers library, considering the sheer size of the dataset.

Conclusion and Takeaways

In this blog post, we took a deep dive into the Hugging Face Tokenizers library, with a focus on the BertWordPieceTokenizer. We covered the following key points:

• Tokenization is a crucial pre-processing step in NLP that converts raw text into a numerical representation suitable for deep learning models.
• The Hugging Face ecosystem provides several popular libraries for working with Transformer models, including the Tokenizers library for state-of-the-art tokenization.
• The Tokenizers library offers a fast and efficient implementation of various tokenization algorithms, including WordPiece, BPE, and SentencePiece.
• The BertWordPieceTokenizer is the default tokenizer used for the BERT model and uses a subword tokenization scheme to handle large vocabularies.
• To use the BertWordPieceTokenizer, you need to load a pre-trained WordPiece vocabulary and call the encode() method on input text.
• The Tokenizers library supports saving and loading trained tokenizer models, encoding pre-tokenized text, and efficient batch encoding of large datasets.
• Benchmarking the BertWordPieceTokenizer on the WikiText-103 dataset showcases the impressive speed of the Tokenizers library, taking just a few minutes to process over 100 million tokens.

The Hugging Face Tokenizers library is a powerful tool in the NLP practitioner‘s toolkit. Its optimized implementation and wide range of supported tokenization algorithms make it suitable for processing text data of all sizes. The ability to save and load trained tokenizers adds flexibility and reusability to NLP workflows.

Whether you‘re working on a language modeling task, fine-tuning a pre-trained Transformer model, or exploring a new NLP problem, the Tokenizers library can help streamline your text pre-processing pipeline. Its ease of use and excellent performance make it a go-to choice for many NLP developers and researchers.

If you‘re interested in leveraging state-of-the-art NLP techniques in your own projects, I highly recommend checking out the Hugging Face Tokenizers library. The official documentation provides a comprehensive guide to getting started and exploring the various features of the library. You can also find a collection of pre-trained tokenizer models in the Hugging Face Model Hub to experiment with.

I hope this deep dive into the BertWordPieceTokenizer has piqued your interest in the Tokenizers library and motivated you to explore its capabilities further. As always, the best way to learn is to get your hands dirty and start coding. Happy tokenizing!