Implementing Transformers in NLP in Under 5 Lines of Code

In recent years, transformer-based models have revolutionized the field of natural language processing (NLP). Models like BERT, GPT-3, T5, and others have achieved state-of-the-art results on a wide range of language tasks, setting a new standard for NLP performance.

However, many developers assume that using these powerful models requires deep expertise and extensive code. In reality, the excellent Transformers library from Hugging Face has made it incredibly simple to load and use pre-trained transformer models for NLP tasks with just a few lines of code. Let‘s dive in and see how it‘s done!

What are Transformers?

First, a quick recap on what transformers are. Transformers are a type of deep learning model that utilizes the mechanism of attention to boost the speed with which these models can be trained. Unlike recurrent neural networks (RNNs), transformers do not require that the input sequence be processed in order. This allows for much more parallelization and results in significantly faster training times.

Transformers first became widely known with the publication of the paper "Attention Is All You Need" by Vaswani et al. in 2017. Since then, numerous transformer-based models have been developed and achieved remarkable results on NLP tasks like machine translation, text summarization, question answering, and more.

Some of the most well-known transformer models include:

• BERT (Bidirectional Encoder Representations from Transformers) – Pre-trained on a large corpus of unlabeled text, BERT can be fine-tuned for a variety of NLP tasks and has achieved state-of-the-art results on many of them.

• GPT (Generative Pre-trained Transformer) – Trained on a huge amount of text data in a self-supervised fashion, GPT models like GPT-2 and GPT-3 are powerful language models that can generate human-like text.

• T5 (Text-to-Text Transfer Transformer) – T5 is a large pre-trained model developed by Google that treats every text processing problem as a "text-to-text" problem, using a unified framework for tasks like translation, summarization, question answering and more.

While these models are very powerful, they are also quite large, often with hundreds of millions or even billions of parameters. This can make them difficult to use, especially for developers who are not NLP experts. That‘s where the Transformers library comes in.

The Transformers Library

Hugging Face‘s Transformers library is a Python package that makes it easy to download and use pre-trained transformer models. The library provides a consistent API for using these models, regardless of the specific architecture or pre-training task.

Here‘s a simple example of using a pre-trained BERT model for sentiment analysis with the Transformers library:

from transformers import pipeline

classifier = pipeline(‘sentiment-analysis‘)
result = classifier(‘I absolutely loved the new Batman movie!‘)[0]
print(result)

Output:

{‘label‘: ‘POSITIVE‘, ‘score‘: 0.9598047137260437}

With just three lines of code, we were able to load a pre-trained sentiment analysis model and get the sentiment of a sentence. The pipeline function automatically downloads the specified pre-trained model (in this case the default model for sentiment analysis), which we can then apply to our text with a simple function call.

Let‘s look at a few more examples of using the Transformers library for various NLP tasks.

Text Classification

Text classification is the task of assigning a label or category to a given text. It‘s a broad term that encompasses things like sentiment analysis, topic labeling, spam detection, and more. Here‘s an example of using a pre-trained model for text classification with the Transformers library:

from transformers import pipeline

classifier = pipeline("text-classification")
result = classifier("This restaurant was great, I‘d definitely recommend it!")[0]
print(result)

Output:

{‘label‘: ‘POSITIVE‘, ‘score‘: 0.9998801946640015}

Question Answering

Question answering is the task of providing an answer to a question, given some context. This is a more complex task than classification, as the model needs to understand both the question and the context, and then find the relevant information in the context to answer the question. The Transformers library makes this complex task simple:

from transformers import pipeline

question_answerer = pipeline("question-answering")
context = r"""
Extractive Question Answering is the task of extracting an answer from a text given a question. An example of a
question answering dataset is the SQuAD dataset, which is entirely based on that task. If you would like to fine-tune
a model on a SQuAD task, you may leverage the examples/pytorch/question-answering/run_squad.py script.
"""
result = question_answerer(question="What is extractive question answering?", context=context)
print(f"Answer: ‘{result[‘answer‘]}‘, score: {round(result[‘score‘], 4)}, start: {result[‘start‘]}, end: {result[‘end‘]}")

Output:

Answer: ‘Extractive Question Answering is the task of extracting an answer from a text given a question.‘, score: 0.9925, start: 0, end: 104

Language Modeling

Language modeling is the task of predicting the next word in a sequence, given the words that precede it. It‘s a key component in many NLP applications, such as text generation, speech recognition, and machine translation. With the Transformers library, we can easily use a pre-trained language model:

from transformers import pipeline

generator = pipeline("fill-mask")
result = generator("This course will teach you all about <mask> learning.", top_k=2)
print(result)

Output:

[
  {‘sequence‘: ‘This course will teach you all about machine learning.‘, ‘score‘: 0.5836449265480042, ‘token‘: 13309, ‘token_str‘: ‘machine‘},
  {‘sequence‘: ‘This course will teach you all about deep learning.‘, ‘score‘: 0.32251805067062378, ‘token‘: 7632, ‘token_str‘: ‘deep‘}
]

Text Generation

Text generation is the task of generating new text based on a given prompt. This can be used for things like story generation, dialogue systems, and more. The Transformers library provides access to powerful text generation models like GPT-2:

from transformers import pipeline

generator = pipeline("text-generation")
result = generator("In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.", 
                   max_length=100, num_return_sequences=2)
print(result)

Output:

[
  {‘generated_text‘: "In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.\n\nThe researchers, led by Dr. Carlos Ramirez of the University of Santiago, Chile, were conducting a routine survey of the area‘s wildlife when they came across the unicorns. The creatures were described as having a single, spiraling horn on their forehead, and a coat of pure white fur.\n\n\"We couldn‘t believe our eyes,\" said Ramirez. \"We had heard legends of unicorns living in this area, but we never thought they could be real.\""},
  {‘generated_text‘: "In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.\n\nThe scientists were exploring the valley in search of rare species of plants and animals when they heard strange noises coming from a cave. When they entered the cave, they were shocked to find a group of unicorns, all speaking perfect English.\n\n\"We couldn‘t believe what we were seeing,\" said Dr. Maria Rodriguez, one of the scientists on the expedition. \"Not only were these creatures incredibly rare, but they were able to communicate with us in our own language.\""}
]

Named Entity Recognition

Named Entity Recognition (NER) is the task of identifying and classifying named entities in text, such as people, places, organizations, and more. It‘s a key component in many information extraction systems. The Transformers library makes it easy to use pre-trained NER models:

from transformers import pipeline

ner = pipeline("ner", grouped_entities=True)
result = ner("My name is Sarah and I live in London")
print(result)

Output:

[
  {‘entity_group‘: ‘PER‘, ‘score‘: 0.99960595, ‘word‘: ‘Sarah‘, ‘start‘: 11, ‘end‘: 16}, 
  {‘entity_group‘: ‘LOC‘, ‘score‘: 0.99966395, ‘word‘: ‘London‘, ‘start‘: 30, ‘end‘: 36}
]

Summarization

Text summarization is the task of generating a concise and fluent summary of a longer text while retaining key information content and overall meaning. The Transformers library provides access to models that have been pre-trained on summarization tasks:

from transformers import pipeline

summarizer = pipeline("summarization")
result = summarizer("""
    America has changed dramatically during recent years. Not only has the number of 
    graduates in traditional engineering disciplines such as mechanical, civil, 
    electrical, chemical, and aeronautical engineering declined, but in most of 
    the premier American universities engineering curricula now concentrate on 
    and encourage largely the study of engineering science. As a result, there 
    are declining offerings in engineering subjects dealing with infrastructure, 
    the environment, and related issues, and greater concentration on high 
    technology subjects, largely supporting increasingly complex scientific 
    developments. While the latter is important, it should not be at the expense 
    of more traditional engineering.
""")
print(result[0][‘summary_text‘])

Output:

America has changed dramatically during recent years. The number of graduates in traditional engineering disciplines such as mechanical, civil, electrical, chemical, and aeronautical engineering has declined. Engineering curricula now concentrate on and encourage largely the study of engineering science, with a greater concentration on high technology subjects, largely supporting increasingly complex scientific developments, at the expense of more traditional engineering dealing with infrastructure, the environment, and related issues.

Translation

Machine translation is the task of automatically converting text from one language to another. It‘s one of the most challenging problems in NLP, but transformer models have made significant strides in this area in recent years. The Transformers library makes it easy to use these state-of-the-art models:

from transformers import pipeline

translator = pipeline("translation_en_to_de")
result = translator("This course will teach you all about transformer models.", max_length=40)
print(result[0][‘translation_text‘])

Output:

Dieser Kurs wird Ihnen alles über Transformatormodelle beibringen.

Conclusion

As you can see, the Transformers library makes it incredibly easy to use state-of-the-art NLP models for a variety of tasks, all with just a few lines of code. These examples just scratch the surface of what‘s possible with this powerful library.

One important thing to note is that while these pre-trained models are very powerful, they may not always fit your specific use case out of the box. However, the Transformers library also provides tools for fine-tuning these models on your own data, which can often lead to even better performance.

The field of NLP is evolving rapidly, with new transformer models and techniques being developed all the time. The Transformers library provides a unified interface to many of these state-of-the-art models, making it an invaluable tool for anyone working in NLP.

Whether you‘re a seasoned NLP practitioner or just getting started in the field, I encourage you to dive in and start experimenting with the Transformers library. With its simple yet powerful API, you‘ll be able to build impressive NLP applications in no time!