A Comprehensive Guide to Image Preprocessing with Scikit-Image

Image preprocessing is a critical yet often overlooked step in computer vision and image-based machine learning. At a high level, preprocessing refers to any transformations or modifications we make to images prior to feeding them into a model or algorithm. This can include resizing, cropping, color adjustments, filtering, normalization, and a variety of other techniques.

The goal of preprocessing is to simplify and standardize image data in a way that makes it easier for models to learn meaningful patterns and achieve strong predictive performance. Careful preprocessing can help improve accuracy, reduce computational overhead, and increase the robustness of computer vision systems to real-world variations in lighting, pose, scale, and other factors.

While the advent of deep learning has automated many aspects of feature engineering, preprocessing remains an important part of the workflow for most computer vision projects. In this guide, we‘ll take a comprehensive look at image preprocessing through the lens of the scikit-image library in Python. We‘ll cover essential concepts and techniques, dive into the latest research and best practices, and provide practical tips for integrating preprocessing into your own machine learning pipelines.

Evolution of Image Preprocessing

Image preprocessing has a long history dating back to the early days of computer vision in the 1960s. Early techniques focused on simple transformations like resizing, cropping, and histogram equalization to improve the consistency and interpretability of image data.

In the 1970s and 80s, researchers began developing more sophisticated methods for edge detection, noise reduction, and image restoration. Filters like the Sobel and Canny detectors allowed for reliable extraction of shape and structure information. Fourier analysis provided a framework for manipulating images in the frequency domain.

The 1990s and 2000s saw the rise of feature-based methods like SIFT and HOG that aimed to represent images in terms of local patterns invariant to changes in scale and rotation. Preprocessing was used to normalize orientation and illumination to improve the reliability of feature matching. Techniques like bag-of-words and spatial pyramids allowed for aggregating local features into global image representations.

In the deep learning era, many classic preprocessing techniques have been replaced by learned feature extractors in the form of convolutional neural networks (CNNs). However, basic transformations like resizing and normalization are still commonly used to ensure consistent input sizes and ranges. Augmentation strategies that randomly perturb images have also become a key preprocessing step for training more robust models.

Resizing Images

Resizing is perhaps the most common preprocessing operation, used to scale images to a consistent size for training and inference. The scikit-image library provides the skimage.transform.resize function for this purpose:

from skimage.transform import resize

resized_image = resize(image, (256, 256))

This resizes the input image to 256×256 pixels using bilinear interpolation by default. We can specify different interpolation methods using the order parameter:

resized_image = resize(image, (256, 256), order=0) # nearest neighbor
resized_image = resize(image, (256, 256), order=3) # bicubic 
resized_image = resize(image, (256, 256), order=5) # Lanczos

The choice of interpolation method can have a significant impact on the quality of the resized image. In general, higher-order methods like bicubic and Lanczos will produce smoother results, while lower-order methods like nearest neighbor are faster but may introduce artifacts.

A study by Ye et al. (2018) compared the performance of different interpolation methods on a range of computer vision tasks. They found that bicubic interpolation generally provided the best tradeoff between speed and accuracy, outperforming bilinear on most benchmarks.

When resizing images, it‘s also important to consider the aspect ratio – the proportional relationship between width and height. Naively resizing to a square aspect ratio can distort features and hurt model performance. To preserve aspect ratios, we can first crop the image to a desired shape or pad it with empty pixels.

Image Normalization

Another key preprocessing step is to normalize pixel intensities to a consistent range, typically between 0 and 1. This helps stabilize gradients during training and allows models to converge faster. Scikit-image provides utility functions for common normalization schemes:

from skimage import img_as_float

normalized_image = img_as_float(image)

The img_as_float function converts images to 64-bit floating point values scaled between 0 and 1. We can also normalize to other ranges like -1 to 1 using img_as_float32 and img_as_float64.

For images with multiple color channels, it‘s important to normalize each channel independently to account for differences in illumination and contrast. A common approach is to subtract the mean and divide by the standard deviation of each channel:

from skimage import img_as_float

normalized_image = img_as_float(image)
means = normalized_image.mean(axis=(0, 1))
stds = normalized_image.std(axis=(0, 1))
normalized_image = (normalized_image - means) / stds

This scales each channel to have zero mean and unit variance. Some studies have found that this type of per-channel normalization can improve CNN accuracy by up to 10% on benchmarks like ImageNet.

Color Conversions

Many computer vision algorithms operate on grayscale or single-channel images to reduce computational complexity. Scikit-image provides functions for converting between color spaces:

from skimage.color import rgb2gray

grayscale = rgb2gray(image)

The rgb2gray function converts an RGB image to grayscale using a weighted combination of the color channels that mimics human brightness perception. We can also convert to other color spaces like HSV and LAB:

from skimage.color import rgb2hsv, rgb2lab

hsv = rgb2hsv(image)  
lab = rgb2lab(image)

The choice of color space can impact downstream tasks like segmentation and object detection. HSV is often used for color-based segmentation, while LAB is perceptually uniform and useful for color transfer and denoising.

Some specialized applications may require converting to non-RGB spaces. For example, many medical imaging modalities like MRI and CT scans are grayscale by default. Satellite images are often multispectral, with channels corresponding to different wavelengths beyond visible light.

In a study of colorectal polyp detection, Shin et al. (2018) found that converting endoscopy images from RGB to HSV and extracting the hue channel improved CNN accuracy by highlighting the polyp regions more strongly.

Image Augmentation

Data augmentation has become a key preprocessing technique for training deep learning models that are more robust to real-world variations. The idea is to programatically generate new training examples by applying random transformations to the original images.

Some common augmentation techniques include:

• Random cropping and resizing
• Horizontal and vertical flips
• Rotations and shears
• Color jittering (brightness, contrast, saturation, hue)
• Adding Gaussian noise
• Applying random filters (blur, sharpen, etc.)

Scikit-image provides a flexible set of tools for implementing these augmentations:

from skimage.transform import rotate, resize
from skimage.util import random_noise
import numpy as np

def augment(image):
    # Random rotation
    angle = np.random.uniform(-20, 20)
    rotated = rotate(image, angle)

    # Random noise
    noise_factor = np.random.uniform(0, 0.05)
    noisy = random_noise(rotated, mode=‘gaussian‘, var=noise_factor)

    # Random resizing
    scale = np.random.uniform(0.8, 1.2)
    resized = resize(noisy, output_shape=image.shape[:2]*scale, mode=‘reflect‘)

    # Random crop
    y, x = resized.shape[:2]
    startx = np.random.randint(0, x - image.shape[1])
    starty = np.random.randint(0, y - image.shape[0])
    cropped = resized[starty:starty+image.shape[0], startx:startx+image.shape[1]]

    return cropped

This function applies a series of random augmentations to an input image, including rotation, Gaussian noise, resizing, and cropping. We can apply it to each image in our training set to generate a augmented version on the fly:

from skimage import io

image = io.imread(‘image.jpg‘)
augmented_image = augment(image)

In practice, augmentation is often implemented using tools like Keras‘ ImageDataGenerator or PyTorch‘s transforms. These allow for defining custom augmentation pipelines that are applied automatically during training.

Augmentation has been shown to significantly improve the accuracy and generalization of deep learning models. In a landmark study, Krizhevsky et al. (2012) used aggressive augmentation including random crops, flips, and color jittering to achieve state-of-the-art results on the ImageNet classification benchmark.

More recent work has explored learned augmentation policies that are optimized for specific datasets and models. For example, AutoAugment (Cubuk et al., 2019) uses reinforcement learning to find optimal sequences of augmentation operations.

Preprocessing Best Practices

Based on a survey of top-performing entries in computer vision competitions on Kaggle and other platforms, some common preprocessing best practices include:

• Resize images to a consistent size that balances computational cost and information retention. Popular choices are 224×224, 256×256, and 299×299.
• Use bilinear or bicubic interpolation for resizing. Avoid nearest neighbor unless computational constraints require it.
• Normalize pixel values to a standard range, typically 0-1 or -1 to 1. Perform per-channel normalization for color images.
• Convert to grayscale or other simplified color spaces when color information is less relevant.
• Apply data augmentation aggressively during training to improve model robustness. Use a combination of geometric and photometric transformations.
• Cache preprocessed images to disk in a format like HDF5 to avoid redundant computation.
• Use transfer learning and pretrained models when possible to reduce the need for extensive preprocessing.

It‘s also important to be mindful of the ethical implications of preprocessing. Techniques like resizing and normalization can amplify biases present in the original data. For example, models trained primarily on light-skinned faces may perform poorly on darker skin tones if the training data is not carefully balanced and preprocessed.

Conclusion

Image preprocessing is a vital part of any computer vision workflow. Scikit-image provides a powerful set of tools for performing essential preprocessing tasks like resizing, normalization, color conversion, and augmentation in Python.

Choosing appropriate preprocessing techniques requires careful consideration of the specific requirements and constraints of your application. It‘s important to balance computational efficiency with preserving relevant information in the images.

Ultimately, the goal of preprocessing is to transform raw image data into a representation that is optimally suited for a given machine learning task. By leveraging techniques like resizing, normalization, and augmentation, we can train models that are more accurate, robust, and efficient.

As computer vision continues to advance, preprocessing will remain an active area of research and development. Promising future directions include learning-based preprocessing pipelines, adaptive preprocessing for transfer learning and domain adaptation, and preprocessing techniques tailored for specialized data types like medical and satellite imagery.

Regardless of the specific techniques used, the principles of simplicity, standardization, and robustness should guide preprocessing choices. By being deliberate and strategic about preprocessing, you can unlock the full potential of your image data and take your computer vision projects to the next level.