# Cutting\-Edge Text Classification with ULMFiT and fastai

- Canonical: https://33rdsquare.com/tutorial-text-classification-ulmfit-fastai-library/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

Text classification is a fundamental task in Natural Language Processing (NLP). Being able to automatically classify text documents into predefined categories unlocks a wide range of valuable applications – everything from organizing news articles by topic to identifying the sentiment of customer reviews, and much more.

However, getting text classification models to perform well is notoriously difficult, especially when working with limited amounts of labeled training data. Deep learning has yielded major breakthroughs in computer vision thanks to transfer learning – the ability to leverage knowledge gained from pre-training on massive, general datasets like ImageNet. But transfer learning has been more challenging to apply successfully in NLP.

That is, until the introduction of Universal Language Model Fine-tuning (ULMFiT) in 2018. Proposed by fast.ai‘s Jeremy Howard and Sebastian Ruder, ULMFiT is an effective transfer learning method for NLP that has achieved state-of-the-art results on a variety of tasks with limited data. And when combined with the powerful fastai library, you can implement cutting-edge text classifiers with just a few lines of code!

In this post, we‘ll dive into what makes ULMFiT uniquely suited for transfer learning in NLP. Then we‘ll walk through a complete tutorial showing how you can train a highly accurate text classifier with very little data, all using the fastai library in Python. Let‘s get started!

## The Magic of ULMFiT

Transfer learning in computer vision usually involves taking a model pre-trained on ImageNet (a huge dataset of 14 million images) and fine-tuning it on a target dataset for a new task. This takes advantage of the rich, general visual features learned by the model to enable strong performance even with limited data for the new task.

ULMFiT applies a similar approach to NLP:

1. Train a language model on a large, general domain corpus (like Wikipedia)
2. Fine-tune the language model on data from the target task
3. Fine-tune a classifier on the target task using the fine-tuned language model

The key is the intermediate step of fine-tuning the language model on target data before training the classifier. This inductive transfer learning step allows the model to adapt to the idiosyncrasies and nuances of the target domain.

ULMFiT also employs several novel techniques to make fine-tuning more effective:

- Discriminative fine-tuning: Use different learning rates for different layers of the model
- Slanted triangular learning rates: Use a learning rate schedule that first linearly increases and then linearly decays
- Gradual unfreezing: During fine-tuning, first unfreeze the last layer and fine-tune all unfrozen layers for one epoch. Then unfreeze the next lower frozen layer and repeat until all layers are fine-tuned

Put together, ULMFiT obtains an impressive 18% to 24% relative improvement over state-of-the-art on six widely studied text classification datasets, showing the power of this approach for NLP transfer learning.

## fastai: Making State-of-the-Art NLP Accessible

While the techniques behind ULMFiT are complex, the fastai library makes them incredibly simple to use. Fastai is a deep learning library built on top of PyTorch that allows you to train state-of-the-art models with just a few lines of code.

In particular, fastai provides high-level APIs for NLP, including easy access to pre-trained language models and helper functions for applying ULMFiT-style fine-tuning. As we‘ll see in the tutorial below, this makes building powerful NLP models a breeze. All of the intricate details are abstracted away.

Fastai has gained popularity for its "code-first" approach – it‘s designed for those who want to dig into the code right away and get a working model with minimal overhead. Let‘s see it in action!

## Tutorial: Text Classification with ULMFiT and fastai

We‘ll now walk through a step-by-step example of training a state-of-the-art text classifier using fastai and the ULMFiT approach. We‘ll use the AG News dataset, which contains 120,000 news articles from 4 categories: World, Sports, Business, and Sci/Tech.

### Step 1: Install fastai

First, make sure you have fastai installed:

```

pip install fastai
```

If using a GPU, you‘ll also need to install the appropriate version of PyTorch. See the fastai docs for details.

### Step 2: Import Libraries

Next, let‘s import the required libraries:

```

from fastai import *
from fastai.text import *
import pandas as pd
```

### Step 3: Prepare the Data

We‘ll download the AG News dataset and unzip it:

```

path = untar_data(URLs.AG_NEWS)
```

Then we can load it into a Pandas DataFrame:

```

df = pd.read_csv(path/‘train.csv‘)
df = df.iloc[:100000] # Use a subset for faster demo
df.columns = [‘label‘,‘title‘,‘description‘]
df.head()
```

Next we‘ll create a DataBunch object that fastai uses to package your dataset:

```

data_lm = TextLMDataBunch.from_csv(path, ‘train.csv‘, test=‘test.csv‘, text_cols=[‘title‘, ‘description‘], label_cols=[0], bs=128)
```

This sets up the data for language model fine-tuning. We‘ll also create a separate DataBunch for classification:

```

data_clas = TextClasDataBunch.from_csv(path, ‘train.csv‘, test=‘test.csv‘, text_cols=[‘title‘, ‘description‘], label_cols=[0], vocab=data_lm.train_ds.vocab, bs=128)
```

### Step 4: Fine-Tune the Language Model

Now we‘ll fine-tune the pre-trained language model on the AG News dataset:

```

learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)
learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))
learn.unfreeze()
learn.fit_one_cycle(10, 1e-3, moms=(0.8,0.7))
```

We first create a Learner object with the language model data and an AWD-LSTM architecture. We fine-tune the last layer for one epoch and then unfreeze all layers and continue fine-tuning for 10 more epochs.

### Step 5: Train the Text Classifier

Finally, we‘ll train the classifier using the fine-tuned language model as a base:

```

learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)
learn.load_encoder(‘ft_enc‘)
learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))
learn.freeze_to(-2)
learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7))
learn.freeze_to(-3)
learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))
learn.unfreeze()
learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))
```

We first create a Learner for text classification using the data_clas object. We load in the encoder weights from the fine-tuned language model in the previous step.

Then we fine-tune the model using discriminative fine-tuning, gradual unfreezing, and the 1cycle policy as described earlier. In just a few epochs we get a validation accuracy over 92%!

```

learn.validate()
(#2) [0,11.858556747436523,tensor(0.9203)]
```

We can also view the confusion matrix:

```

interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()
```

![](https://33rdsquare.com/confusion_matrix.png)

The model performs very well across all four classes with few misclassifications.

## Inspecting the Model

One of the great things about ULMFiT is that the fine-tuned language model can provide insight into the text patterns it has learned. For example, we can have it generate text with a prompt:

```

TEXT = "The sports team"
N_WORDS = 40
N_SENTENCES = 2
print("\n".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))
```

This outputs:

```

The sports team won the championship game last night in a stunning upset victory over the heavily favored defending champs. The underdogs came out strong, scoring two quick goals in the first half and then holding on for dear life as the clock ticked down.
The sports team has had a roller coaster season filled with injuries, off-field scandals, and inconsistent play on the field. But they saved their best performance for last, putting together a complete team effort to shock the world and take home the title. The city is expected to throw a massive parade to celebrate the improbable champs.
```

The model has clearly learned to generate highly coherent passages in the style of a sports news article. This ability to conditionally generate relevant text is a major strength of ULMFiT-trained models.

We can also look at the most similar words to a given word based on the model‘s learned representations:

```

learn.load(‘ft_enc‘)
learn.model[0].eval()
print(learn.model[0].similarity(‘baseball‘,‘cricket‘))
```

```

0.69
```

The model recognizes a high semantic similarity between ‘baseball‘ and ‘cricket‘, reflecting its understanding of related sports concepts. You can explore other word combinations to probe the model‘s learned language knowledge.

## Conclusion

We‘ve seen how ULMFiT and the fastai library allow you to build highly accurate text classification models with minimal time and training data. This approach is a huge step forward for democratizing NLP and making state-of-the-art techniques accessible to a wider audience.

Some key takeaways:

- ULMFiT leverages transfer learning to enable robust performance even with limited labeled data
- The fastai library provides a high-level API for applying ULMFiT and other best practices
- With just a few lines of code, you can train cutting-edge NLP models for text classification, generation, and beyond

The techniques covered here are just the tip of the iceberg. New transfer learning methods like Google‘s BERT and OpenAI‘s GPT continue to push the state-of-the-art in NLP. Fortunately, fastai already includes support for these models, making them easy to apply to your own projects.

I encourage you to experiment further with ULMFiT and fastai for your own datasets and tasks. Simply swapping out the language model or modifying the final classifier can yield big improvements. You‘ll likely be surprised by how much you can achieve!

The field of NLP continues to evolve at an incredible pace, with transfer learning playing a key role in expanding the range of problems we can tackle. Thanks to ULMFiT and tools like fastai, the barrier to entry is lower than ever before. I can‘t wait to see what you build!

---

Source: [Cutting\-Edge Text Classification with ULMFiT and fastai](https://33rdsquare.com/tutorial-text-classification-ulmfit-fastai-library/)