Supercharging Your NLP Models with Text Data Augmentation: A Guide to Using TextAttack

Data augmentation has become an essential tool in the arsenal of NLP practitioners looking to improve the performance of their models, especially when faced with the all-too-common challenge of limited labeled training data. By generating additional training examples from existing data, augmentation can significantly boost model accuracy and generalization. In this guide, we‘ll explore text data augmentation techniques and demonstrate how to effectively apply them using the TextAttack library.

The Many Levels of Text Augmentation

Text data augmentation techniques can be applied at multiple levels of granularity:

1. Character level – Modifying characters within words, such as through random insertions, deletions or substitutions.

2. Word level – Replacing individual words with synonyms, hypernyms, or other semantically similar words.

3. Phrase level – Paraphrasing parts of a sentence or merging/splitting sentences.

4. Document level – Modifying the entire text, such as by round-trip translation or generating new text conditioned on the original.

Choosing the right level(s) to apply augmentation depends on your specific NLP task and data. For most text classification tasks, word and phrase level techniques are most commonly used.

Powerful Augmentation Techniques

Let‘s take a closer look at some of the most effective text data augmentation methods:

Easy Data Augmentation (EDA)

EDA is a set of simple yet surprisingly effective augmentation techniques proposed in a 2019 paper by Wei and Zou. These include:

• Synonym Replacement (SR): Randomly choosing words in a sentence and replacing them with synonyms.
• Random Insertion (RI): Finding synonyms for a random word in a sentence and inserting them into a random position.
• Random Swap (RS): Randomly choosing two words in a sentence and swapping their positions.
• Random Deletion (RD): Randomly removing words from a sentence with a pre-defined probability.

The genius of EDA lies in its simplicity. These techniques can be easily applied using basic NLP tools like WordNet, and have been shown to boost performance across a range of text classification tasks.

Back-Translation

Back-translation is a clever augmentation method that leverages the imperfect nature of machine translation (MT) to paraphrase text. It works by:

1. Translating a text from language A to language B using an MT model
2. Translating the result back into language A using another MT model

Since MT is not perfect, the twice-translated text will usually be different than the original while preserving its overall semantics. Back-translation can generate diverse paraphrases using different source-target language pairs.

Conditional Text Generation

Large pre-trained language models like GPT-3, BERT, and T5 can be used for data augmentation by generating new text conditioned on the class label in addition to the original text. For example, a BERT-based conditional generator could be fine-tuned to take an input text and target class, and generate a new semantically similar example for that class. This leverages the language model‘s ability to produce fluent text while allowing control over the augmented example‘s label.

Augmentation with TextAttack

Now that we‘ve covered some key augmentation techniques, let‘s see how we can easily apply them using the excellent TextAttack library. TextAttack provides a suite of augmentation methods, as well as adversarial attacks, for text.

We‘ll use TextAttack‘s 6 built-in augmenters to generate augmented examples for a movie review:

from textattack.augmentation import WordNetAugmenter, EmbeddingAugmenter, EasyDataAugmenter, CharSwapAugmenter, CheckListAugmenter, CLAREAugmenter

text = "This movie was awesome! The acting was incredible and the plot kept me on the edge of my seat the whole time."

print("Original:", text)

for augmenter in [WordNetAugmenter(), EmbeddingAugmenter(), EasyDataAugmenter(), CharSwapAugmenter(), CheckListAugmenter(), CLAREAugmenter()]:
    augmented_text = augmenter.augment(text)
    print(f"\n{augmenter.__class__.__name__}:", augmented_text[0])

This outputs:

Original: This movie was awesome! The acting was incredible and the plot kept me on the edge of my seat the whole time.

WordNetAugmenter: This movie was awesome! The acting was incredible and the plot kept me on the edge of my seat the whole prison term.

EmbeddingAugmenter: This movie was awesome! The acting was amazing and the plot kept me on the edge of my seat the whole time.

EasyDataAugmenter: This movie was aw3som3 ! The acting was incredible and the plot kept me on the edge of my seat the whole time.  

CharSwapAugmenter: This movie was awesome! The acting was incredible and th eplot kept me on the edge of my seat the whole time.

CheckListAugmenter: This film was awesome! The acting was incredible and the plot kept me on the edge of my seat the whole time.  

CLAREAugmenter: This movie was awesome! The acting was outstanding and the plot kept me on the edge of my seat the whole time.

As you can see, each augmenter applies different transformations while preserving the original sentiment and much of the core semantics. The WordNetAugmenter replaces "time" with the synonym "prison term". The EmbeddingAugmenter substitutes "incredible" with the semantically similar "amazing". EasyDataAugmenter performs a random character substitution in "awesome". CharSwapAugmenter swaps the space and "p" in "the plot". CheckListAugmenter replaces "movie" with the synonym "film". And CLAREAugmenter generates a new fluent sentence with "incredible" replaced by "outstanding".

Measuring the Impact of Augmentation

To see the impact of data augmentation, let‘s compare the accuracy of a sentiment analysis model trained with and without augmentation on the SST-2 dataset. We‘ll use the EmbeddingAugmenter to double the size of the training data.

First, let‘s train the model without augmentation:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from datasets import load_dataset

dataset = load_dataset(‘sst2‘)
train_texts, train_labels = dataset[‘train‘][‘text‘], dataset[‘train‘][‘label‘] 
test_texts, test_labels = dataset[‘validation‘][‘text‘], dataset[‘validation‘][‘label‘]

model = Pipeline([
    (‘tfidf‘, TfidfVectorizer()),
    (‘clf‘, LogisticRegression()) 
])

model.fit(train_texts, train_labels)
baseline_accuracy = model.score(test_texts, test_labels)
print(f"No Augmentation Accuracy: {baseline_accuracy:.2%}")

No Augmentation Accuracy: 81.67%

Now let‘s augment the training data and re-train:

from textattack.augmentation import EmbeddingAugmenter

augmenter = EmbeddingAugmenter()

aug_train_texts, aug_train_labels = train_texts.copy(), train_labels.copy()
for text, label in zip(train_texts, train_labels):
    aug_text = augmenter.augment(text)[0]
    aug_train_texts.append(aug_text)
    aug_train_labels.append(label)

model.fit(aug_train_texts, aug_train_labels)
aug_accuracy = model.score(test_texts, test_labels)
print(f"Augmentation Accuracy: {aug_accuracy:.2%}")
print(f"Improvement: {aug_accuracy - baseline_accuracy:.2%}")

Augmentation Accuracy: 83.21%
Improvement: 1.54%

With synonym replacement using the EmbeddingAugmenter, test accuracy improved by over 1.5% absolute! This demonstrates the potential of augmentation to boost performance, especially with relatively simple techniques.

Benefits of Text Data Augmentation

As we‘ve seen, text data augmentation can provide several key benefits:

1. Improved model accuracy and generalization – Augmentation generates additional training examples which can significantly improve model performance and reduce overfitting. This is especially valuable when labeled data is scarce.

2. Reduced data collection and labeling costs – Augmentation allows you to get more value from your existing labeled data, reducing the need for expensive manual labeling.

3. Addressing class imbalance – Datasets often have imbalanced class distributions, which can hinder model training. Targeted augmentation of minority class examples can help balance the distribution.

4. Increased model robustness – Augmentation introduces variability in the training data, which can improve a model‘s robustness to noise, typos, and other variations it may encounter in the wild.

Conclusion

In this guide, we‘ve seen the significant impact data augmentation can have on boosting NLP model performance. Libraries like TextAttack make it incredibly easy to apply a variety of augmentation techniques to your text data.

Whether you‘re working with customer reviews, social media posts, or legal documents, text data augmentation is a powerful tool to improve your models. The next time you‘re faced with limited training data, give augmentation a try – you might be surprised at the results!

The field of text data augmentation is rapidly evolving, with new techniques regularly being proposed. Some exciting areas to watch include large language model-based approaches, as well as learned augmentation policies. It‘s an exciting time to be an NLP practitioner, and data augmentation is a key technique to have in your toolbox.

So go forth and augment your text! Your models will thank you.