Harnessing the Power of Preprocessing Layers in TensorFlow Keras

Introduction

When building machine learning models, one of the most critical steps is preparing your data correctly. Raw data often comes in various formats, ranges, and distributions that are not optimal for training neural networks. This is where data preprocessing comes into play. Preprocessing transforms the raw input data into a representation that is more amenable to learning by the model.

Fortunately, if you‘re using TensorFlow with Keras, a lot of the heavy lifting of data preprocessing can be handled efficiently using built-in preprocessing layers. In this article, we‘ll take a deep dive into TensorFlow‘s Keras preprocessing layers. You‘ll see how you can leverage these layers to streamline your data pipeline, by integrating preprocessing directly into your model architecture. This leads to cleaner, more portable, and easier to deploy end-to-end models.

We‘ll cover the different types of preprocessing layers available, look at code examples of how to use them, and discuss best practices and tips to get the most out of Keras‘ preprocessing functionality. Whether you‘re working with structured data, unstructured text, or images, preprocessing layers can radically simplify your workflow. Let‘s jump in and preprocess some data!

The Importance of Data Preprocessing

Before we examine the specifics of preprocessing layers in Keras, it‘s worth taking a step back and considering why data preprocessing matters in the first place. Many raw datasets contain noise, outliers, missing values, and features at different scales. Attempting to train a model directly on messy, unprocessed data is likely to lead to suboptimal performance.

Some key objectives of data preprocessing include:

• Cleaning: Removing or imputing missing data, dealing with outliers and errors
• Normalization: Transforming numeric features to have similar scales and distributions
• Encoding: Converting categorical variables to numeric representations
• Enrichment: Deriving new, more informative features from the raw data
• Augmentation: Synthesizing new training examples to improve model robustness

Traditionally, much of the data preprocessing is done using libraries like pandas or scikit-learn before the data is fed to the model. However, this approach has some drawbacks. There‘s potential for training/serving skew if the preprocessing isn‘t exactly replicated when the model is deployed. It also clutters the model building process with a lot of data transformation logic.

Keras preprocessing layers address these issues by moving the preprocessing into the model itself. The big advantage is that the exact same preprocessing will be executed whether you are training the model or using it for inference later. Preprocessing layers are also built on top of TensorFlow ops, so they can benefit from hardware acceleration and be used when exporting models to different runtimes.

Overview of Keras Preprocessing Layers

Keras offers a variety of preprocessing layers designed to operate on different types of input data. They can be used just like any other layer in a Keras Sequential or Functional model. Some of the common preprocessing layers include:

• Normalization: Rescale and standardize numeric features
• TextVectorization: Convert raw text into an encoded representation
• StringLookup / IntegerLookup: Map strings or integers to categorical encodings
• Resizing / CenterCrop: Resize and crop images to a target size
• Rescaling: Scale image pixel values, e.g. from [0, 255] to [0, 1]

The general workflow with preprocessing layers is:

1. Instantiate the preprocessing layer and set its parameters
2. Adapt the layer to a sample of the training data to learn statistics
3. Incorporate the layer into your model architecture
4. The layer will preprocess data automatically during model training and inference

One of the great things about preprocessing layers is that they make your TensorFlow models more portable and reusable. The preprocessing becomes an integral part of the model, so you can easily export a full inference pipeline using tf.saved_model. This makes preprocessing easy to replicate if deploying the model to a mobile app using TensorFlow Lite or a web app using TensorFlow.js for example.

Let‘s take a closer look now at some of the most useful Keras preprocessing layers and see how you can leverage them for your own models.

Normalization and Feature Scaling

One of the most basic but important types of preprocessing is feature normalization. Many ML algorithms perform better when all the input features are on a similar scale and distribution. The Normalization layer in Keras implements feature-wise normalization of input data.

To use the Normalization layer, you first adapt it to a sample of your data. This allows the layer to learn the mean and variance of each feature. It uses this information to standardize the input features by subtracting the mean and dividing by the standard deviation. Here‘s an example:

from tensorflow.keras.layers import Normalization

data = np.array([[0.1, 0.5, 1.2], 
                 [-0.3, -1.5, 2.2],
                 [1.1, 0.8, -0.4]])

normalizer = Normalization(axis=-1)
normalizer.adapt(data)

normalized_data = normalizer(data)
print("Mean:", normalized_data.numpy().mean())
print("Variance:", normalized_data.numpy().var())

This will apply a Z-score normalization to each feature, transforming them to have 0 mean and unit variance. You can also configure the Normalization layer to rescale features to a specific range, e.g. between -1 and 1, by setting the output_min and output_max parameters.

For more control over the scale and distribution of each numeric feature, you can use a discretization or binning approach. The Discretization layer allows you to chunk continuous features into discrete buckets. This turns the numeric data into categorical data which can then be encoded. Configure the number of bins and strategy for the cutpoints between bins.

from tensorflow.keras.layers import Discretization

data = np.array([[-1.5, 2.2, 3.1], 
                 [0.2, -0.5, 0.3]])

discretizer = Discretization(num_bins=4, epsilon=0.01)
discretizer.adapt(data)

discretized_data = discretizer(data)

Text Vectorization

When working with text data, deep learning models require the raw text to be converted to numeric representations like integer sequences or embeddings. The Keras TextVectorization layer provides an easy way to preprocess raw text into an encoded format that can be fed directly to an Embedding layer or Dense layer.

The TextVectorization layer will tokenize the input text and build a vocabulary of the most frequent tokens. You can configure things like the vocabulary size, whether to strip punctuation, and the output sequence length. The layer is adapted to a sample of the text data, allowing it to build the vocab. It can then transform text examples into an encoded representation.

from tensorflow.keras.layers import TextVectorization

text_data = [
    "The quick brown fox jumps over the lazy dog.",
    "The five boxing wizards jump quickly."
]

vectorizer = TextVectorization(max_tokens=10, output_mode=‘int‘, output_sequence_length=6)
vectorizer.adapt(text_data)

vectorized_text = vectorizer(text_data)
print(vectorized_text)

The output_mode can be set to ‘int‘ to encode the text as a sequence of token indices, ‘binary‘ for one-hot encoding, ‘count‘ for token counts, or ‘tf-idf‘. By default the layer will generate variable-length encodings but you can use the output_sequence_length argument to produce fixed-length sequences through padding or truncation.

You can use the TextVectorization layer in conjunction with lookup layers like StringLookup or IntegerLookup to map the encoded token indices to external vocabularies. This allows you to do things like map tokens to pre-trained word embeddings.

Image Preprocessing

For image data, the Keras preprocessing layers can handle a lot of common transformations. Some standard operations are resizing, cropping, and pixel normalization. These are often needed to get images into a shape the model expects and to improve convergence of the model.

The Resizing and CenterCrop layers transform the width and height dimensions of input images. Resizing will resize the image to a target height and width, using bilinear interpolation by default. CenterCrop slices the central portion of the image, which is useful if you want to crop out the surrounding pixels.

from tensorflow.keras.layers import CenterCrop, Resizing

image_input = tf.keras.Input(shape=(None, None, 3))
cropped = CenterCrop(height=150, width=150)(image_input)
resized = Resizing(height=256, width=256)(cropped)

The Rescaling layer is used to scale the pixel values of input images. For example, to go from pixel values between 0 and 255 to a 0 to 1 floating point scale. This is a common preprocessing step that accelerates model training. You can also incorporate image augmentation layers to synthetically expand the training set with label-preserving transforms.

from tensorflow.keras import Sequential
from tensorflow.keras.layers import Rescaling, RandomFlip, RandomRotation

data_augmentation = Sequential([
    Rescaling(scale=1./255),
    RandomFlip("horizontal"),
    RandomRotation(factor=0.02),
])

Categorical Feature Encoding

If your data contains string-valued or high-cardinality categorical features, you‘ll typically want to encode them to an integer or vector representation. Keras provides a few layers designed to make it easy to preprocess categorical features inline with your model.

For features that can take on a fixed set of values, like a categorical product type or user nationality, the StringLookup and IntegerLookup layers allow you to encode input strings or integers into a one-hot or multi-hot representation that can be ingested by a neural network. Adapt the layer on the training data to build a vocabulary of unique values the feature can take. Then use this to encode feature values as vectors.

from tensorflow.keras.layers import StringLookup

vocab = [‘cat‘, ‘dog‘, ‘fox‘]

lookup = StringLookup(vocabulary=vocab, output_mode=‘one_hot‘)

data = tf.constant([‘dog‘, ‘cat‘, ‘bird‘])
encoded = lookup(data)

For very high cardinality features like user ids or zip codes, where the number of unique values the feature can take is too large to one-hot encode, you can use the Hashing layer. This applies a hash function to the input values and encodes the hash as a vector of a desired size. Using a fixed output vector size allows you to handle sparse, high-dimensional categorical inputs without needing to maintain a giant vocabulary. The hash encoding can be fed into an Embedding layer to learn a dense representation.

from tensorflow.keras.layers import Hashing

hashing = Hashing(num_bins=10)

data = tf.constant([‘id_79352‘, ‘id_24091‘, ‘id_12993‘])
hashed = hashing(data)

Integrating Preprocessing Layers into Models

To get the full benefit of Keras‘ preprocessing layers, you‘ll want to integrate them directly into your model definitions. You can think of a Keras model as an end-to-end pipeline encompassing the complete feature preprocessing, feature generation, and model prediction flow. This makes your models highly portable since the necessary preprocessing is packaged with the model artifact.

For example, here‘s how you could build an image classification model with integrated preprocessing:

from tensorflow.keras import Sequential 
from tensorflow.keras.layers import Rescaling, RandomFlip, RandomRotation
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Flatten

image_input = tf.keras.Input(shape=(256, 256, 3))
x = Rescaling(scale=1./255)(image_input)
x = RandomFlip("horizontal")(x)
x = RandomRotation(factor=0.02)(x)
x = Conv2D(filters=32, kernel_size=(3, 3), activation=‘relu‘)(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Conv2D(filters=64, kernel_size=(3, 3), activation=‘relu‘)(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Flatten()(x)
x = Dense(units=64, activation=‘relu‘)(x)
x = Dense(units=10, activation=‘softmax‘)(x)

model = tf.keras.Model(inputs=image_input, outputs=x)

With this approach, calling model.predict() on a raw input image will automatically execute the complete pipeline of preprocessing and inference. When you save the model using model.save(), the preprocessing layers get saved as well.

Best Practices and Tips

Here are a few recommendations to keep in mind when using Keras preprocessing layers:

• Get to know the data specs the different preprocessing layers expect and output. Refer to the API docs to understand how they handle things like variable-length and rank > 1 inputs.

• Remember that the layers with vocabularies (TextVectorization, StringLookup, etc.) always need to be adapted to data before use. Fit on a representative sample of training examples.

• The Normalization and Discretization layers can be adapted to the mean and variance of your complete dataset for more representative statistics, unlike layers with vocabularies which should only be adapted to training data.

• The Keras Preprocessing Layers are most effective when incorporated directly into your model graphs. This keeps feature preprocessing logic tightly integrated with the model.

• For large datasets, adapt preprocessing layers using the tf.data API in dataset mode, which can handle datasets that don‘t fit in memory. Use adapt(train_data) instead of adapt(train_data.batch(batch_size)).

• Most preprocessing layers are compatible with tf.distribute for training models across multiple GPUs or machines. Wrap the layer creation and model compiling in a strategy.scope().

• To handle variable length text sequences, specify output_mode="int" and output_sequence_length=None in TextVectorization so that it returns ragged tensors natively. Use this with a Keras Embedding layer and a recurrent neural network.

What‘s New in TensorFlow 2.x

The Keras preprocessing layers are under active development so stay tuned for new functionality! Some recent enhancements in TensorFlow 2.x include:

• Updates to TextVectorization layer: The TextVectorization layer now supports custom string standardization and splitting functions, as well as improved handling of variable-length output sequences through ragged tensors (enabled by default).

• Improved preprocessing support on GPU: Many of the preprocessing ops like hashing, vocabulary lookups and normalization can now be GPU-accelerated on supported hardware. The preprocessing layers will automatically place eligible ops on GPU when available.

• New Discretization and Hashing layers: The Discretization layer provides a way to turn continuous numerical features into integer-encoded categorical features by binning. The Hashing layer allows high-cardinality categorical features to be encoded to fixed-length vectors using a hash function.

It‘s a great time to start leveraging the power of Keras preprocessing layers in your TensorFlow 2.x projects!

Conclusion

In this article, we took a close look at the awesome functionalities provided by the preprocessing layers in the TensorFlow Keras API. These layers greatly simplify the process of getting your data ready for deep learning models. Text, images, numerical data, categorical features – there‘s a preprocessing layer for that!

The key takeaway is that preprocessing layers allow you to integrate the full preprocessing pipeline into your model graphs, leading to cleaner, more modular, and easier to deploy models. Especially for use cases like mobile or web deployments using TensorFlow Lite or TensorFlow.js, having preprocessing baked into your model makes things much more streamlined.

I encourage you to try out Keras preprocessing layers the next time you‘re building a TensorFlow model. Think about how to combine various preprocessing layers into an end-to-end pipeline encompassing the journey from raw data to model predictions. With some practice, Keras preprocessing layers will become an indispensable tool in your deep learning toolbox. Happy preprocessing!