Deploying Machine Learning Models as APIs with FastAPI and Heroku

Deploying machine learning models into production as reliable, scalable APIs is a core skill for ML engineers and data scientists. By wrapping models in web services, they become much more useful and accessible – other systems can now consume their predictions!

There are many ways to achieve this, but this post will focus on one compelling approach: using the Python web framework FastAPI to build prediction APIs and deploying them easily to the Heroku cloud platform. We‘ll walk through a detailed example to understand the key concepts and steps.

Why Deploy Models as APIs?

Before we dive in, let‘s consider the benefits of exposing machine learning models through web APIs:

Reproducibility: Serving models as APIs ensures they are used in a consistent way by clients. Once an API contract is defined, the same inputs will always produce the same outputs. This isn‘t necessarily the case with other patterns like embedded models.

Separation of Concerns: Deployed models can be developed, tested and maintained independently from the various applications that consume them. This decoupling makes both systems simpler.

Scalability: Once a model is deployed as an API, we can utilize all the usual tools and patterns for scaling web services to handle large request volumes. Strategies like load balancing, autoscaling, and serverless can be applied.

Flexibility: Clients written in any programming language can consume models exposed as API endpoints. This is more flexible than language-specific deployment options.

Monitoring: With models behind API endpoints, we can monitor their performance and use with standard web monitoring tools. Request logging, error rates, latency percentiles, and traffic patterns all become observable.

So in summary, deploying models as APIs makes them more reproducible, scalable, flexible and observable. The trade-off is that it does add some complexity over simpler deployment schemes.

The FastAPI Framework

To turn our model into an API, we‘ll use the FastAPI web framework. FastAPI is a modern, fast, Python 3.6+ framework for building APIs. It has quickly gained popularity due to its speed, ease of use, and powerful feature set.

Here are some notable features of FastAPI:

• High Performance: FastAPI leverages Starlette to achieve high performance (on par with NodeJS and Go). The framework was designed for fast execution.

• Easy to Use: FastAPI uses standard Python type hints for declaration and validation. Its design promotes reuse and minimizes boilerplate code.

• Automatic API Documentation: Interactive API documentation and exploration web interfaces are automatically generated and included by default. This makes it easy to visualize, understand and test your API endpoints.

• Async Support: FastAPI supports asynchronous request handling via the async/await syntax. You can write async handlers without complicated threading or event loops.

• Serialization: Data validation and serialization are based on Python type hints using the Pydantic library. This allows for clear, expressive APIs and provides automatic type checking.

• Dependency Injection: FastAPI has a powerful dependency injection system that makes it easy to declare resources needed by routes and have them automatically injected.

• Security: The framework has built-in support for security and authentication standards like OAuth2 and JWT tokens.

• Database Integration: FastAPI works well with database libraries like SQLAlchemy, Peewee, and Tortoise ORM. You can use Pydantic models in your API schemas that integrate seamlessly.

But most importantly for our purposes, FastAPI is an excellent choice for building machine learning APIs. It provides a simple, intuitive way to define API endpoints, request/response schemas, and serve predictions from models.

Preparing a Model for Deployment

Now let‘s get practical and walk through the steps to deploy an ML model API with FastAPI. The complete code is available in this GitHub repository.

The first step is preparing your trained model to be deployed. This involves saving your model in a format that can be efficiently loaded and used for inference in a production API setting.

There are several things to consider when saving models for deployment:

• Serialization format: Popular options include pickling, HDF5, TensorFlow SavedModel, and ONNX. The choice depends on your ML library and production constraints.

• Versioning: Including the model version as part of the saved file name is a good practice. This makes it easy to track and rollback models in production.

• Environment consistency: It‘s important to use the same dependencies (Python version, library versions, etc.) when loading the model in production as when the model was trained and saved. Docker containers are useful for maintaining this consistency.

• Testing: Before deploying, you should load and test your saved model in an environment as close to production as possible. Validate that it performs as expected on hold-out data.

For this example, we‘ll use a simple scikit-learn random forest model that predicts iris flower species from petal and sepal measurements. The model is trained on the classic Iris dataset.

Here‘s the code that trains and saves the model:

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
import pickle

# Load dataset 
iris = load_iris()
X, y = iris.data, iris.target

# Train model
X_train, X_test, y_train, y_test = train_test_split(X, y)
clf = RandomForestClassifier()
clf.fit(X_train, y_train)

# Serialize model
with open("iris_rf.pkl", "wb") as f:
    pickle.dump(clf, f)

This saves the random forest model to a pickle file iris_rf.pkl. We‘ll now use this saved model file in our FastAPI application.

Creating the FastAPI App

Let‘s create the FastAPI app that will load this serialized model and use it to serve predictions.

Here‘s the outline of main.py:

from fastapi import FastAPI
from pydantic import BaseModel
import pickle

app = FastAPI()

class IrisInput(BaseModel):
    sepal_length: float
    sepal_width: float
    petal_length: float
    petal_width: float

class IrisOutput(BaseModel):
    species: str

with open("iris_rf.pkl", "rb") as f:
    model = pickle.load(f)

@app.post("/predict", response_model=IrisOutput)
def predict(iris: IrisInput):
    input_data = iris.dict()
    input_vector = [[input_data[‘sepal_length‘], input_data[‘sepal_width‘], 
                     input_data[‘petal_length‘], input_data[‘petal_width‘]]]

    prediction = model.predict(input_vector)[0]
    species = iris.target_names[prediction]

    return IrisOutput(species=species)

Let‘s break this down step-by-step:

1. We import the necessary libraries: FastAPI for the web framework, BaseModel from Pydantic for input/output schemas, and pickle for loading the model.

2. We create an instance of the FastAPI class called app. This will be the main point of interaction for our API.

3. We define Pydantic models that specify the input fields needed for making a prediction (IrisInput) and the format of the API response (IrisOutput). These models define the API contract.

4. We load the pickled model file and bind the resulting Python object to the name model.

5. We define the /predict API endpoint using the @app.post decorator. This tells FastAPI that the decorated function should be executed whenever a POST request is sent to the /predict path.

6. Inside the prediction function, we extract the input data from the IrisInput model. Since our model expects a 2D array, we reshape the input values.

7. We use the loaded model to generate a prediction from the input data. The model.predict() function returns an integer representing the predicted class index.

8. We look up the predicted class name using the target_names mapping provided by scikit-learn.

9. We return an IrisOutput instance containing the predicted species. FastAPI will automatically serialize this to JSON in the response body.

That‘s the complete API implementation in about 20 lines of code! Let‘s test it out.

Testing the API

To test our API locally, first install the dependencies:

pip install fastapi uvicorn scikit-learn

Then start the FastAPI server:

uvicorn main:app --reload

You should see output like:

INFO:     Started server process [28720]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

Now let‘s make a test prediction request using curl:

curl -X POST http://127.0.0.1:8000/predict \
    -H ‘Content-Type: application/json‘ \
    -d ‘{"sepal_length": 5.1, "sepal_width": 3.5, "petal_length": 1.4, "petal_width": 0.2}‘

{"species":"setosa"}

Our API successfully loads the model, makes a prediction, and returns it in the expected JSON format. We‘re ready to deploy it!

Deploying to Heroku

For deploying our model API, we‘ll use Heroku, a popular platform-as-a-service. Heroku makes it easy to deploy Python web applications with minimal configuration.

To deploy, you‘ll need a free Heroku account and the Heroku CLI installed. Follow these steps:

1. Create requirements.txt:

   fastapi
   uvicorn
   gunicorn
   scikit-learn

2. Create a Procfile to tell Heroku how to run the app:

   web: gunicorn -w 4 -k uvicorn.workers.UvicornWorker main:app

3. Set up a Git repository and commit your code:

   git init
   git add .
   git commit -m "Initial commit"

4. Create a new Heroku app:

   heroku create

5. Push your code to Heroku:

   git push heroku main

Heroku will detect that this is a Python application, install the dependencies from requirements.txt, and start the app according to the Procfile.

After the build process completes, you‘ll see the URL where your API is deployed:

remote: -----> Launching...
remote:        Released v3
remote:        https://mighty-springs-67739.herokuapp.com/ deployed to Heroku

Let‘s test our deployed API by making another prediction request, this time to the Heroku URL:

curl -X POST https://mighty-springs-67739.herokuapp.com/predict \
    -H ‘Content-Type: application/json‘ \
    -d ‘{"sepal_length": 7.0, "sepal_width": 3.2, "petal_length": 4.7, "petal_width": 1.4}‘

{"species":"versicolor"}

It works! We now have a production-ready, publicly accessible API for our machine learning model. This pattern of serving models through APIs can be applied to a wide range of ML/AI systems.

Considerations for ML Model Deployment

We walked through a simple example of deploying an ML model API, but there are additional considerations when putting models into production. Let‘s touch on a few key points:

Model Storage: In this example, we loaded the serialized model from a file in the same directory as our code. For larger models or multiple models, it‘s common to use cloud storage services like Amazon S3 or Google Cloud Storage. The model files can be downloaded at application startup.

Inference Speed: Depending on your model‘s size and complexity, generating predictions could become a bottleneck under high request volumes. There are several ways to optimize serving performance, including using faster model formats like ONNX, leveraging caching, and employing model compression techniques.

GPU Inference: For deep learning models, performing inference on a GPU can dramatically improve throughput. FastAPI applications can be configured to run on a machine with a GPU and use libraries like TensorFlow or PyTorch for hardware-accelerated inference.

Scaling: As your API traffic increases, you‘ll likely need to scale your infrastructure. With Heroku, you can easily add more dynos to run additional instances of your application. More advanced scaling techniques like horizontal scaling and distributed serving may be needed for very high-traffic systems.

API Versioning: If your model will be updated over time, it‘s important to version your API. This allows clients to continue using the existing API when you deploy an updated model. FastAPI supports API versioning through the URL path, query parameters, or headers.

Monitoring: Once deployed, you‘ll want to keep a close eye on your model API. Key metrics to track include request rate, error rate, latency, and inference performance. Tools like Prometheus and Grafana are commonly used for API monitoring.

Logging: Robust logging is essential for troubleshooting issues with your deployed model API. FastAPI uses standard Python logging, which can be configured to record detailed information about each request and response. The logs can be aggregated and analyzed using a centralized logging solution.

Security: When deploying an ML API, security is paramount. API authentication and rate limiting help prevent misuse of your model. Additionally, you should validate and sanitize all incoming data to protect against adversarial attacks. FastAPI provides several tools for implementing secure APIs.

While we can‘t cover all of these topics in depth here, it‘s important to be aware of them when planning to deploy ML models at scale. As with all software systems, architecting for production requires careful design and testing.

Conclusion

In this post, we explored how to deploy machine learning models as web APIs using the FastAPI framework and Heroku. We walked through an end-to-end example of taking a trained scikit-learn model, creating an API endpoint to serve predictions, and deploying the application.

To recap, the key steps are:

1. Train and save your ML model
2. Create a FastAPI application
3. Define input/output schemas with Pydantic
4. Load the serialized model
5. Implement the prediction endpoint
6. Test the API locally
7. Deploy to Heroku

We also touched on several architectural considerations for deploying ML models in production, including model storage, inference optimization, scaling, versioning, monitoring, logging, and security.

Deploying models via APIs built with FastAPI and Heroku is a powerful design pattern for integrating machine learning into production systems. The frameworks abstract away many of the complexities of model serving infrastructure, allowing data scientists and ML engineers to focus on building accurate, reliable models.

I hope this post has been a helpful introduction to the world of ML model deployment. The complete code is available on GitHub. Happy deploying!