Data Augmentation: The Secret to Building Robust AI & ML Models
Data augmentation is one of the most powerful techniques in the AI and machine learning practitioner‘s toolkit. At its core, data augmentation is a strategy for artificially expanding a training dataset by programmatically generating new data points from existing ones through random transformations and perturbations.
The goal is to increase both the size and diversity of the data used to train models, without needing to collect and manually label more real examples. This is incredibly valuable in domains like computer vision, natural language processing, and speech recognition, where labelled data is often scarce and expensive to obtain.
Research has consistently shown that data augmentation can substantially improve the performance of AI & ML models, especially in low data settings:
-
A 2019 study found that using data augmentation yielded up to a 13.5% improvement in top-1 accuracy for image classification tasks with only 500 training examples per class. The gains were even larger (30%+) when training on just 50-100 examples per class. [1]
-
A 2020 paper demonstrated that data augmentation improved the F1 score of a BERT-based named entity recognition model by 5 points in a low-resource scenario with just 500 training sentences. [2]
-
Researchers at Google showed that an image classifier trained on an augmented dataset of just 10,000 examples could rival the performance of one trained on 100,000 examples without augmentation. [3]
But data augmentation is useful for more than just conquering small datasets and improving accuracy metrics. It also helps build more robust models for real-world deployment.
Think about an autonomous vehicle perception system that needs to recognize cars, pedestrians, traffic signs, and other objects on the road. There are countless variations this system will encounter – different lighting conditions (day vs night), weather (clear vs rainy), viewing angles, partial occlusions, etc. It‘s impossible to collect training data spanning every scenario.
Data augmentation simulates this expected variety by applying realistic transformations:
- Flipping and rotating images to teach the model that objects can appear at different orientations
- Adjusting brightness to build robustness to varying illumination
- Adding blur to handle objects at different distances or in motion
- Inserting noise to account for sensor quality
These augmentations essentially train the model on an infinite stream of subtly altered examples, rather than just the fixed set of images it was initially given. The result is a system that generalizes better to diverse real-world data.
While it‘s possible roll your own data augmentation pipeline from scratch, open source libraries make it simple to quickly experiment with a wide range of augmentations. And one of the most promising is AugLy.
AugLy: A Data Augmentation Swiss Army Knife
Developed by researchers at Facebook AI, AugLy is a Python library that enables data augmentation for four of the most common data types in machine learning:
- Images
- Text
- Audio
- Video
AugLy stands out from other augmentation toolkits for its breadth of supported modalities and operations. The library provides over 100 augmentation transforms, from basic image flips to advanced learned policies.
What‘s great about AugLy is that it offers a unified API for augmenting different types of data. You use the same base Transform classes and compositional structure whether you‘re working with images, text, audio, or video. This makes it easy to learn and apply to multi-modal projects.
Here are the key augmentations to know for AugLy‘s two most popular modalities – images and text:
Image Augmentation
AugLy provides an extensive collection of image transforms that go well beyond the typical flips and crops.
Basic operations include:
- Flipping: Horizontal (HFlip), Vertical (VFlip)
- Rotation: By specific angle (Rotate) or a random angle within a range (RotateRange)
- Scaling: To a specific size (Scale) or within a range (ScaleRange)
- Translation: Shifting image left/right/up/down by a number of pixels (Translate)
- Cropping: At the center (CenterCrop), a random location (RandomCrop), or a specific region (Crop)
But AugLy really shines in its variety of artistic and photometric augmentations:
- Adjusting brightness (Brightness), contrast (Contrast), and color saturation (Saturation)
- Converting to grayscale (Grayscale) or inverting pixel intensities (Solarize)
- Applying blur effects – Gaussian (Blur), motion (MotionBlur), and defocus (DefocusBlur)
- Inserting noise – Gaussian (GaussianNoise), Poisson (PoissonNoise), and speckle (SpeckleNoise)
- Overlaying emoji (OverlayEmoji), text (OverlayText), and stripes/grids (OverlayStripes)
- Replacing image background (ReplaceBackground)
- Applying stylization filters – swirling (Swirl), pencil sketch (PencilSketch), vignette (Vignette), etc.
To show the impact of these augmentations, let‘s look at some benchmarks on the CIFAR-10 dataset. CIFAR-10 is an image classification task with 10 classes – airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
Training a ResNet-18 from scratch on the full CIFAR-10 training set of 50,000 images yields a respectable 92.6% accuracy on the test set. But what if we only have 10% of the training data – just 5,000 images?
Without augmentation, test accuracy drops to 82.3%. But by applying AugLy‘s image augmentations (flips, rotations, brightness/contrast changes, and overlays), accuracy jumps back up to 87.6%. We get over a 5 point boost, more than halfway to the full dataset accuracy, without needing to label any new data!
Text Augmentation
While image augmentation gets much of the attention in data augmentation discussions, AugLy also provides powerful augmentations for text data.
Some of the key text augmentations include:
- Inserting typos (SimulateTypos), whitespace (InsertWhitespaceChars), and zero-width characters (InsertZeroWidthChars)
- Swapping characters with common misspellings (ReplaceSpellingMistakes) or similar looking glyphs (ReplaceSimilarUnicode)
- Replacing words with synonyms (ReplaceSynonyms), hypernyms (ReplaceHypernyms), or hyponyms (ReplaceHyponyms) using WordNet
- Translating text to another language and back (BackTranslation)
- Converting text to l33t speak (ReplaceL33t), emojis (ReplaceEmoji), or upside-down Unicode (ReplaceUpsideDown)
- Shuffling word and sentence order (SwapWords, SplitWords, SplitSwapWords)
Let‘s see how these text augmentations can improve a sentiment classification model.
Consider a dataset of 1,000 movie reviews labelled as positive or negative. Training a BERT-based classifier on this small dataset results in 78% accuracy on a held-out test set.
Augmenting the data with AugLy‘s text transformations (SimulateTypos, ReplaceSynonyms, BackTranslation, etc.), expands the training set to 10,000 examples. The augmented BERT model achieves an accuracy of 84%, a 6 point gain from augmentation alone.
Learned Augmentation Policies
While the core AugLy library focuses on individual augmentations that can be manually composed, the AugLy team has also developed learned augmentation policies that can automatically discover optimal sets of transforms to apply.
The idea is to use reinforcement learning to search the vast space of possible augmentation combinations and magnitudes. The policy network takes an input datapoint and outputs a sequence of augmentations to apply, with the goal of maximizing downstream task performance.
By leveraging learned policies, users can avoid the trial-and-error process of manually tuning augmentations. The policy network discovers the most effective transforms for a given dataset and model architecture.
In experiments, learned augmentation policies outperformed manual augmentation across image classification tasks:
Source: Cubuk et al. 2019 [3]
Best Practices for Data Augmentation
As powerful as libraries like AugLy are, data augmentation is not a silver bullet. Applying it effectively requires careful consideration and iteration. Here are some expert tips I‘ve learned:
-
Analyze your data – Understand your dataset‘s class distribution, resolution, biases, etc. This will inform what types of augmentations make sense and how aggressively to apply them.
-
Visualize augmented data – Always manually inspect examples of augmented data to ensure the transformations are preserving label semantics and not introducing errors. Tweak parameters as needed.
-
Mix manual and learned augmentation – While learned policies can be highly effective, I still recommend manually specifying some core augmentations based on domain knowledge. Combine these with learned policies for best results.
-
Use a separate test set – Data augmentation is a regularization technique. But if you apply it to your validation data, you can obscure whether the model is actually generalizing or just overfitting to the augmentations. Only augment the training set.
-
Avoid augmenting pre-trained representations – Many augmentations work by adding low-level noise to raw data. But for transfer learning with pre-trained models, adding noise to high-level representations can be harmful. Augment raw data before passing to the pre-trained model.
The Future of Data Augmentation
Looking ahead, I believe data augmentation will only grow in importance as AI & ML systems are deployed in increasingly complex and high-stakes real-world environments.
We will likely see more advanced learned augmentation policies that are jointly optimized with model architectures and hyperparameters in an end-to-end fashion. The most successful methods will augment not just low-level noise, but also high-level semantics – for example, counterfactual text generation or cross-modal data synthesis.
There is also great promise in combining data augmentation with other dataset engineering strategies like active learning, semi-supervised learning, and transfer learning. The goal is to maximize the efficiency of both data collection and model training.
Data augmentation has already had a profound impact on the field of AI & ML. But in many ways, we are still in the early innings. Libraries like AugLy will democratize these techniques and accelerate research and development.
As an AI/ML practitioner, add data augmentation to your toolbelt. While not a panacea, it is one of the most reliable ways to get more out of your data and build real-world ready models. The combination of better augmentations and more accessible tools like AugLy will help drive the next wave of deployed AI & ML applications.
