Master Image Augmentation with These Powerful Python Libraries

Image augmentation is a crucial technique in modern computer vision and deep learning workflows. By programmatically generating new training examples from existing images, augmentation helps improve the robustness and generalization of our models.

With the right augmentation strategy, we can substantially increase the diversity of our training data without having to collect and manually label more images. This is especially valuable when training data is limited or expensive to acquire.

Fortunately for Python developers, there are some fantastic open source libraries that make it easy to apply a wide variety of augmentations to our image data. In this post, we‘ll dive deep into three of the most popular and full-featured options:

• Keras ImageDataGenerator
• Albumentations
• AugLy

We‘ll walk through detailed code examples of how to use each library and discuss some of the most effective augmentation techniques. By the end, you‘ll be equipped to skillfully apply augmentation to your own projects and reap the benefits of increased model performance.

Let‘s get started!

Keras ImageDataGenerator

We‘ll begin with the ImageDataGenerator class, which is part of the Keras deep learning library. While not as comprehensive as the other libraries we‘ll cover, it‘s a great starting point given its simplicity and tight integration with Keras‘ modeling APIs.

Here‘s a basic example of how we can use ImageDataGenerator to apply random horizontal flips, rotations, zooms, and shifts to a set of images:

from tensorflow.keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(
        rotation_range=40,
        width_shift_range=0.2,
        height_shift_range=0.2,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True,
        fill_mode=‘nearest‘)

We simply specify the types of augmentations we want and the parameters controlling their intensity. Keras will then take care of generating batches of augmented images for us during training.

For example, we can fit a model using augmented data like so:

model.fit(
      datagen.flow(x_train, y_train, batch_size=32),
      steps_per_epoch=len(x_train) / 32, epochs=epochs)

The flow() method generates batches of augmented images on the fly so we don‘t have to store them all in memory. This makes Keras‘ augmentation very efficient.

While ImageDataGenerator offers a solid set of basic augmentations, it lacks some of the more advanced transforms provided by other libraries. Let‘s take a look at a more powerful alternative next.

Albumentations

Albumentations is an extremely popular augmentation library designed with computer vision tasks in mind. It provides over 70 different augmentations and a simple, yet flexible interface.

Unlike Keras, Albumentations is not tied to any particular deep learning framework. It‘s meant to be a general purpose tool for augmenting image data, regardless of how you plan to consume it downstream.

Here‘s an example of defining a set of augmentations using Albumentations:

import albumentations as A

transform = A.Compose([
    A.CLAHE(),
    A.RandomRotate90(),
    A.Transpose(),
    A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.50, rotate_limit=45, p=.75),
    A.Blur(blur_limit=3),
    A.OpticalDistortion(),
    A.GridDistortion(),
    A.HueSaturationValue(),
])

The Compose class lets us chain together multiple augmentations into a single transformation pipeline. We can then apply this pipeline to any image like so:

transformed = transform(image=image)
transformed_image = transformed["image"]

One of the standout features of Albumentations is its excellent performance. It‘s written in an optimized way to be blazing fast even when applying complex sequences of augmentations. This is critical for speeding up ML workflows.

Some of the noteworthy augmentations provided by Albumentations include:

• Geometrical transforms like rotations, flips, elastic deformations
• Color space transforms like RGB shifts, CLAHE, channel shuffling
• Blurring/sharpening transforms
• Noise injection like Gaussian and ISO noise
• Cutout augmentations like CoarseDropout and GridDropout
• Weather simulations like adding snow, shadows, fog

This extensive set of transforms allows us to realistically simulate many different real-world scenarios and greatly improve our model‘s ability to handle novel images.

AugLy

AugLy is a relatively new library developed by Facebook AI Research. Its goal is to provide a unified interface for accessing a large collection of augmentations sourced from many different libraries.

Rather than re-implementing everything from scratch, AugLy intelligently combines transforms from libraries like Albumentations, Pillow, and OpenCV. This gives us access to an extremely wide range of state-of-the-art techniques.

Here‘s an example of using AugLy to randomly apply one of several possible augmentations to an image:

import augly.image as imaugs

transform = imaugs.RandomAugment(
  transforms=[
    imaugs.ColorJitter(brightness_factor=2.0, 
                        contrast_factor=2.0, 
                        saturation_factor=2.0),
    imaugs.Blur(),
    imaugs.RandomNoise(),
    imaugs.Grayscale(),
    imaugs.RandomRotation(max_degrees=15),
  ]
)

transformed = transform(image)

The RandomAugment function makes it easy to apply one or more randomly selected augmentations to an image. This is a great way to inject variety into our pipeline.

Other interesting augmentations offered by AugLy include:

• OverlayEmoji for pasting emoji graphics onto images
• OverlayText for adding text overlays
• Pixelization for lowering image resolution
• Shuffle for shuffling image patches
• EncodingQuality for simulating poor image encoding
• Instagram filters like 1977 and Brannan
• Artistic filters like Cartoon and Pencil Sketch

AugLy even supports augmentations for audio and text data in addition to images. Its vast collection of transforms make it a one-stop shop for many augmentation needs.

Tips for Effective Augmentation

Now that we‘ve seen how to implement augmentations in code, let‘s discuss some best practices for getting the most out of these techniques:

Tailor augmentations to your data and problem. The best augmentation strategy will vary depending on the nature of your images and what you‘re trying to predict. For example, horizontally flipping images makes sense for general object recognition but not for text recognition. Think carefully about what transformations are likely to appear in your inference data.

Start simple and gradually increase complexity. Tempting as it may be to throw the kitchen sink of augmentations at your model straightaway, it‘s often better to start with a small set of basic transforms and slowly add variety. Inspect augmented images to verify they‘re still recognizable and contain the relevant features. You may find that a relatively modest augmentation strategy works quite well.

Use a mix of domain-relevant transforms. In addition to general-purpose augmentations like crops and flips, consider applying transforms that are representative of your specific data domain. For example, if working with medical images, techniques like MRI simulation or artificially adding tumors could help improve robustness. If classifying manufacturing defects, adding scratches, dents, and smudges to your training images may boost performance.

Beware of augmentation leakage. When splitting data into train/validation/test sets, be sure to apply augmentations only after the split and only to the training set. You don‘t want your model to validate or test on augmented versions of images it trained on, as this can lead to inflated performance metrics.

Search over augmentation parameters. Augmentation techniques usually have hyperpararameters that control their intensity, e.g. rotation angles, noise levels, color shifts. Rather than blindly picking values for these, experiment with different settings to find what works best. You can even use automated hyperparameter optimization to search for an ideal configuration.

Real-World Examples

To further illustrate the power of augmentation, let‘s look at a couple real-world examples of how it‘s been used to achieve state-of-the-art performance on challenging computer vision tasks.

In a 2019 paper, researchers from Google Brain demonstrated how a well-engineered augmentation pipeline could be used to significantly improve performance on the ImageNet object classification benchmark. By combining a set of carefully selected augmentations like MixUp, CutMix, and RandAugment, they were able to train an EfficentNet model to achieve top-1 accuracy of 88.5% on ImageNet. This was a whopping 6.5% improvement over the same model trained without augmentation.

Another powerful example comes from the world of medical imaging. Researchers from NVIDIA developed an augmentation scheme for improving the performance of 2D and 3D MRI scan classification models. By applying transforms that mimic real-world variations in image appearance, like changes in contrast, hue, and saturation, they were able to boost AUC by up to 4% on a brain tumor classification task. Considering the high stakes nature of medical diagnosis, even small performance gains like this can translate to meaningful clinical impact.

These examples underscore that augmentation is much more than a hacky trick for inflating dataset size. When thoughtfully applied, it can be a critical tool for squeezing out performance on even the most intensively studied computer vision problems.

Augmentation as a Path to More Data-Centric AI

A key theme that‘s emerged in the AI community recently is the idea of "data-centric" AI. This is the notion that in many cases, the path to better model performance lies not in engineering newer, more complex architectures, but rather in improving the quality of the data used to train models.

The traditional paradigm has been very model-centric — throwing more computing power and architectural innovations at problems to eke out performance gains. But there‘s a growing recognition that focusing energy on dataset quality via techniques like augmentation, labeling improvements, and data cleaning might provide more bang for the buck.

Indeed, the Google Brain researchers mentioned above found that a model trained with an optimized augmentation strategy could match the performance of a newer, more complex model trained without augmentation while using an order of magnitude less computing resources.

At a time when the computational costs and environmental footprint of massive AI models are skyrocketing, being able to achieve higher performance with less computation by focusing on data quality is immensely appealing. Augmentation is arguably the most important tool in the data-centric AI toolbox.

Conclusion

We‘ve covered a lot of ground in this post, from in-depth usage examples of leading Python augmentation libraries to a discussion of real-world applications and the movement toward data-centric AI.

The key takeaways are:

• Image augmentation is a powerful technique for improving computer vision model performance, especially when training data is limited
• Python libraries like Keras ImageDataGenerator, Albumentations, and AugLy make it easy to apply a wide range of augmentations to image data
• Careful selection and validation of augmentations tailored to your specific data and problem is critical
• Augmentation has enabled state-of-the-art performance on challenging benchmarks and real-world applications
• Focusing on data quality through techniques like augmentation is a promising path to more efficient and sustainable AI progress

If you‘re building computer vision models, investing in augmentation is one of the highest leverage things you can do to boost performance. Experiment with the tools and techniques covered here, and see what they can do for your own projects. A world of augmented possibilities awaits!