Monitoring Airline Delay Prediction Models in Production with Evidently and Streamlit

Introduction

Machine learning models often degrade in performance when deployed to production and faced with live data that differs from what they were trained on. This is a common and critical challenge, as models that performed well in development may make poor predictions on new data, leading to negative business impact.

The solution is to proactively monitor the data and model to detect issues like data drift, target drift, data quality problems, and drops in model performance metrics. By catching these problems early, data scientists and ML engineers can investigate root causes and take corrective actions like retraining the model, fixing data quality problems, or adjusting business logic.

In this post, we‘ll walk through how to implement monitoring for an airline flight delay prediction model using two powerful open-source tools – Evidently and Streamlit. By the end, you‘ll be able to integrate monitoring into your own model development workflow to maintain model performance in production.

Evidently: Open-Source Tool for Analyzing Data and Model Quality

Evidently is an open-source Python library for data scientists and ML engineers to analyze, test, and monitor data and model quality throughout the ML lifecycle. It has three key components:

1. Data Profiling: Generate data profiling reports to understand dataset structure and statistics.

2. Data and Model Checks: Validate data and models against defined checks for data schema, data quality, and model performance.

3. Data and Model Monitoring: Continuously evaluate data and model in production for data drift, data quality issues, target drift, and model performance degradation.

Some of the key benefits of Evidently for monitoring include:

• Interactive visual reports: Evidently generates interactive HTML reports with plots and tables summarizing metrics over time. This makes it easy to spot issues.

• Flexible integrations: Evidently can be integrated into Jupyter notebooks, Python pipelines, or web apps built with tools like Streamlit. This makes it adaptable to different workflows.

• Customizable: While Evidently provides many built-in reports and metrics, you can also customize it to your use case. You can define custom metrics, change thresholds for alerts, and more.

In this tutorial, we‘ll leverage Evidently‘s monitoring capabilities to analyze reference and production data for an airline delay prediction model, generate data and model quality reports, and visualize them in a Streamlit dashboard.

Streamlit: Framework for Building Data Apps

Streamlit is an open-source app framework for building interactive data and ML tools quickly and easily. With just a few lines of Python, you can create a beautiful dashboard that enables users to interact with data, visualize model results, and more. Streamlit takes care of the front-end complexity so you can focus on the data and logic.

Some of the key features and benefits of Streamlit include:

• Simple, clean syntax: Streamlit provides a simple API for defining the UI. You can add text, charts, maps, and interactive widgets in just a few lines of code.

• Fast development: With features like automatic code reloading and built-in caching, Streamlit drastically reduces development time for building data apps.

• Interactivity out-of-the-box: Interactive widgets like sliders, dropdowns, and buttons make it easy to capture user input and update the app in real-time. This is powerful for What-If analyses, etc.

• Flexible deployment: Streamlit apps can be run locally, deployed using platforms like Streamlit sharing and Heroku, or even embedded in existing tools.

We‘ll use Streamlit to build an interactive dashboard to visualize the monitoring reports generated by Evidently. This will make it easy for stakeholders to view model performance over time and enable data scientists to dig deeper into issues.

Step-by-Step Tutorial

Now let‘s dive into the step-by-step implementation! We‘ll walk through loading the data, generating Evidently reports, displaying the results in Streamlit, and interpreting the monitoring insights.

Loading Reference and Production Data

The first step is to load two datasets:

1. Reference data: This is a static dataset that represents the data used for model training and evaluation. It serves as a baseline to compare production data against.

2. Production data: This is time-series data representing live data the model is making predictions on in production. We‘ll load a new batch of production data each time we run our monitoring pipeline.

Here‘s sample code to load the datasets:

import pandas as pd

reference_data = pd.read_csv("reference_data.csv")
production_data = pd.read_csv("production_data.csv")

Generating Evidently Monitoring Reports

With the datasets loaded, we can leverage Evidently to generate monitoring reports. The Evidently Report class provides a simple API to define and run a report analyzing reference and production data.

Here‘s code to generate a report analyzing data drift:

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=reference_data, current_data=production_data)

This code creates a Report object with the DataDriftPreset metric preset, which includes various statistical tests for detecting data drift. The report is then executed by calling run with the reference and production datasets.

We can generate similar reports for other aspects like:

• Data quality: Detect missing values, unexpected categorical levels, abnormal numeric ranges
• Target drift: Analyze changes in the distribution of the target variable
• Model performance: Evaluate predictions vs actuals on metrics like RMSE, MAE, etc.

Visualizing Reports in Streamlit

With the monitoring reports generated, we can visualize them in an interactive Streamlit dashboard. Here‘s skeleton code for the app:

import streamlit as st
from evidently.dashboard import Dashboard

st.set_page_config(layout=‘wide‘)
st.title("Airline Delay Prediction Model Monitoring")

dashboard = Dashboard(tabs=[DataDriftTab(), DataQualityTab(), TargetDriftTab(), RegressionPerformanceTab()])
dashboard.calculate(reference_data, production_data)
dashboard.save("monitoring_dashboard.html")

with open("monitoring_dashboard.html", ‘r‘) as f:
    html_data = f.read()

st.components.v1.html(html_data, width=1200, height=800, scrolling=True)

This code sets up a Streamlit app with a title and creates an Evidently Dashboard with different report tabs. It calculates the reports on the reference and production data and saves the output to an HTML file. Finally, it displays the HTML report in the Streamlit app using the components API.

This results in an interactive dashboard that users can explore to view metrics over time, slice and dice different segments, and dig deeper into potential issues.

Interpreting Monitoring Insights

With the monitoring reports generated and visualized, we can analyze them to derive insights and determine follow-up actions. Here are a few key things to look for:

• Data drift: Are there significant changes in the distribution of important input features between reference and production data? This could signal upstream data changes, data quality issues, or concept drift that may degrade model performance.
• Data quality: Are there columns with lots of missing values, unexpected categories, or abnormal numeric ranges in production? These need to be investigated and fixed in the data pipeline.
• Target drift: Is the distribution of the target variable (flight delay) significantly different in production compared to reference? If so, the model may be making predictions on a different population than it was trained for.
• Model performance: Are key performance metrics like RMSE and MAE significantly worse in production compared to reference? Consider retraining the model or adjusting business logic.

The key is to look for significant deviations in the distributions and metrics between reference and production, and meaningful degradation in performance. Not every change requires action, but major issues should be investigated and resolved to maintain model reliability.

Model Retraining Considerations

An important question is when to retrain the model based on the monitoring insights. While model degradation may signal the need to retrain, it‘s important to consider a few factors before investing the time and effort:

1. Severity and impact of degradation: Is the drop in performance significantly harming key business metrics? Retrain if the model is no longer meeting its intended purpose.

2. Cause of degradation: Is the root cause data drift, data quality issues, or concept drift? If data issues are resolved, the current model may perform fine without retraining. Retrain if the underlying data generating process has fundamentally changed.

3. Quantity and quality of new data: Is there enough new, representative data to retrain on? If only a small amount of production data has been collected, it may be better to wait and collect more before retraining.

4. Cost and complexity of retraining: How expensive is it to retrain in terms of time, resources, and opportunity cost? Weigh the expected benefits of retraining against the costs.

As a general rule, retraining should not be the immediate reaction to minor model degradation. First focus on fixing data issues, then monitor the model‘s performance. If it continues to degrade even with good data, and the impact is significant, then retraining may be warranted.

Conclusion

Monitoring data and model quality in production is critical for maintaining the reliability and performance of machine learning applications. In this post, we walked through how to implement monitoring for an airline flight delay prediction model using Evidently to generate reports and Streamlit to visualize them in an interactive dashboard.

The key steps are:

1. Load a reference dataset and a new batch of production data
2. Use Evidently to generate data drift, data quality, target drift, and model performance reports comparing the two datasets
3. Visualize the reports in a Streamlit dashboard for easy consumption
4. Analyze the reports to detect significant issues and determine corrective actions
5. Consider retraining the model if degradation is severe, the cause is concept drift, and there is sufficient new data

By implementing this monitoring process and considering the nuances around model retraining, data scientists and ML engineers can catch issues early and keep their models performing optimally in production. The end result is greater value for the business and better experiences for end users.

Integrating tools like Evidently and Streamlit into the ML workflow makes monitoring simple and accessible. They provide powerful capabilities out-of-the-box but also customizability to suit a variety of use cases. As you embark on your own model monitoring initiatives, consider how you can adapt this tutorial to your unique context.

Happy monitoring!