Deploying Deep Learning Models to Production with TensorFlow Serving
Developing a high-performing deep learning model is a significant accomplishment, but it‘s only part of the battle. To provide real value, that model needs to be deployed to a production environment where it can be accessed by other systems and generate predictions on new data. This crucial last step of model deployment comes with its own set of challenges and complexities that are often overlooked.
In this post, we‘ll take a deep dive into model deployment for deep learning, focusing on TensorFlow Serving, one of the most popular and powerful tools for deploying TensorFlow models to production. We‘ll cover the key concepts, walk through a hands-on tutorial, and discuss best practices to set you up for success. Let‘s get started!
Why Model Deployment Matters
A machine learning model only provides value when it is deployed in production and being actively used to make predictions on new data to drive decisions and actions. Models that remain in a Jupyter Notebook or only run on your local machine are merely science experiments. Deployment is what turns a model into a useful product or service.
However, model deployment presents a new set of challenges beyond training a model:
- The model needs to be integrated with upstream data sources and downstream systems
- It needs to generate predictions with low latency, often in real-time
- The model needs to scale to handle large volumes of requests
- You need to monitor the model‘s performance and provision resources efficiently
- The model needs to be updatable as new versions are developed
Due to these added complexities, it‘s best to plan for deployment from the start of a project, including choosing appropriate tools and architectures. One key decision is how to deploy your trained model so that it can serve predictions.
Deployment Options for Deep Learning Models
There are a few common approaches for deploying deep learning models to production:
-
Embed the model into your application code and deploy the full application. This is straightforward but tightly couples the model with a specific application, making updates difficult.
-
Deploy the model as a microservice using a web framework like Flask to expose a prediction API. This decouples the model from the application but requires writing custom web serving code.
-
Use a dedicated model serving platform like TensorFlow Serving, Clipper, SageMaker, or KFServing. These handle serving infrastructure for you and provide advanced features like model versioning and rollbacks.
Option 3 is usually the best approach for deep learning models, especially complex ones, and TensorFlow Serving is one of the most mature and widely used tools in this space. Let‘s take a closer look at it.
What is TensorFlow Serving?
TensorFlow Serving is an open-source, high-performance system for serving machine learning models built using TensorFlow. Originally developed by Google, it is designed for production environments and optimized for speed, reliability, and scalability.
Some key features of TensorFlow Serving include:
- Support for multiple model versions and A/B testing
- Batching of requests for efficient serving
- Caching of model instances in memory
- Automatic loading and unloading of models based on usage
- Integration with Kubernetes for scaling and infrastructure management
- Client libraries for multiple languages including Python, Java, and C++
TensorFlow Serving operates as a separate process from your main application. Your application sends prediction requests over gRPC or HTTP/REST to the server, which loads the model, runs inference, and returns the predictions back to the application. Multiple models can be deployed on the same server, each accessible via their own endpoints.
Installing TensorFlow Serving
Before using TensorFlow Serving, you‘ll need to install it on the server where you want to deploy your models. There are a few options for installation:
-
Install using an apt package if using Ubuntu:
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | tee /etc/apt/sources.list.d/tensorflow-serving.list curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add - apt update apt install tensorflow-model-server -
Install using Docker. TensorFlow Serving provides a prebuilt Docker image that you can use:
docker pull tensorflow/serving -
Build from source. This provides the most flexibility but takes the most work. See the TensorFlow Serving documentation for full instructions.
For this tutorial, we‘ll use the Docker image for simplicity.
Deploying a Model using TensorFlow Serving
Now let‘s walk through the process of deploying a trained TensorFlow model using TensorFlow Serving. We‘ll use a simple Keras model that classifies images of clothing from the Fashion MNIST dataset.
First, let‘s train and save the model:
fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
train_images = train_images / 255.0
test_images = test_images / 255.0
model = tf.keras.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation=‘relu‘),
tf.keras.layers.Dense(10)
])
model.compile(optimizer=‘adam‘,
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=[‘accuracy‘])
model.fit(train_images, train_labels, epochs=10)
model.save(‘fashion_model/1‘)
This code trains the model and saves it to a directory called ‘fashion_model/1‘. The ‘1‘ indicates this is version 1 of the model. TensorFlow Serving identifies models by a name and version number.
Next, we need to start the TensorFlow Serving server, pointing it to the directory containing our saved model:
docker run -p 8501:8501 \
--mount type=bind,source=/path/to/fashion_model,target=/models/fashion_model \
-e MODEL_NAME=fashion_model -t tensorflow/serving
This command starts the server, listening on port 8501, and loads the model we just saved. The MODEL_NAME environment variable specifies the name of the model, which clients will use to request predictions.
Now let‘s make a request to the server to get predictions from our deployed model:
import json
import requests
import numpy as np
data = json.dumps({"signature_name": "serving_default", "instances": test_images[0:3].tolist()})
headers = {"content-type": "application/json"}
json_response = requests.post(‘http://localhost:8501/v1/models/fashion_model:predict‘, data=data, headers=headers)
predictions = json.loads(json_response.text)[‘predictions‘]
print(predictions)
This code sends a POST request to the server with a batch of 3 images. The server runs the images through the model and returns the predicted class probabilities. We can see the model is working and our serving infrastructure is set up correctly!
Deploying an Updated Model Version
One of the key benefits of TensorFlow Serving is how easy it makes deploying a new version of a model. To demonstrate this, let‘s make some changes to improve our model, then save it as version 2:
model = tf.keras.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation=‘relu‘),
tf.keras.layers.Dense(128, activation=‘relu‘),
tf.keras.layers.Dense(10)
])
model.compile(optimizer=‘adam‘,
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=[‘accuracy‘])
model.fit(train_images, train_labels, epochs=10)
model.save(‘fashion_model/2‘)
To deploy this new version, we simply add the new model files to a directory named ‘2‘ and restart the TensorFlow Serving container:
docker run -p 8501:8501 \
--mount type=bind,source=/path/to/fashion_model,target=/models/fashion_model \
-e MODEL_NAME=fashion_model -t tensorflow/serving
TensorFlow Serving will automatically detect the new model version and begin using it to serve predictions. We can specify which version we want to use in the client request:
json_response = requests.post(‘http://localhost:8501/v1/models/fashion_model/versions/2:predict‘, data=data, headers=headers)
And that‘s it! No code changes are needed in the client application. We can continue to deploy new versions of the model and run A/B tests between versions to compare performance.
Advanced TensorFlow Serving Features
Beyond the basics we‘ve covered, TensorFlow Serving has a number of advanced features to support more complex deployment scenarios:
- Batching requests together for improved efficiency
- Serving multiple models (with different names) from the same server
- Exporting and serving SavedModel bundles directly from an Estimator
- Integration with Kubernetes for automating model deployment and scaling
See the TensorFlow Serving documentation for details and tutorials on these more advanced use cases.
TensorFlow Serving in the Real World
TensorFlow Serving is used in production at many companies to deploy a wide variety of models, from computer vision to natural language processing to recommendation systems. Some examples include:
- Zendesk uses TensorFlow Serving to deploy NLP models for automatically tagging customer support tickets
- Gannett uses TensorFlow Serving to personalize news article recommendations in real-time
- Airbnb uses TensorFlow Serving to deploy home booking models and hometour recommendation models
As these examples show, TensorFlow Serving is a flexible, battle-tested tool powering machine learning in production at serious scale. For any TensorFlow model that needs to be deployed, it should be one of the top tools you consider.
Wrapping Up and Next Steps
As we‘ve seen, deploying models is a key capability for machine learning engineers, and TensorFlow Serving is a powerful tool to streamline the deployment process for TensorFlow models. We covered the key concepts, walked through a simple model deployment example, and discussed some advanced features and use cases.
To learn more and go deeper with TensorFlow Serving, I recommend checking out the following resources:
- The official TensorFlow Serving documentation
- Google‘s TensorFlow Serving tutorial
- My in-depth course on deploying machine learning models with hands-on projects using TensorFlow Serving
I hope this post has given you a solid foundation for adding TensorFlow Serving to your machine learning toolbelt. Happy deploying!
