The Ultimate Guide to Effortless ML Monitoring with Evidently AI

Introduction

Machine learning models are increasingly being deployed in production to power critical applications and business processes. However, once a model is in production, it‘s not a "set it and forget it" situation. ML models can degrade in performance over time due to data drift, concept drift, and other factors. This can lead to poor user experiences, incorrect predictions, and negative business outcomes if left unchecked.

This is where ML monitoring comes in. ML monitoring is the practice of continuously tracking the performance and behavior of ML models in production to ensure they are functioning as intended. Effective monitoring allows you to proactively identify issues and take corrective action before they cause significant impact.

While ML monitoring is crucial, it can also be challenging, especially for organizations new to operationalizing ML. Some common challenges include:

• Tracking a wide range of metrics across model performance, data quality, resource utilization, etc.
• Detecting hard-to-spot issues like data drift that may degrade model performance gradually over time
• Integrating ML monitoring seamlessly into existing ML pipelines and infrastructure
• Configuring appropriate alerts to notify the right stakeholders when issues arise
• Investigating and troubleshooting model failures to identify the root cause

Fortunately, open-source tools like Evidently AI are making ML monitoring more accessible and easier to implement. In this guide, we‘ll dive deep into how you can leverage Evidently to set up end-to-end ML monitoring for your own projects. Whether you‘re a data scientist, ML engineer, or just getting started with ML, this guide will equip you with the knowledge and code examples to implement effective monitoring and keep your models performing optimally in production.

What is Evidently AI?

Evidently is an open-source Python library for analyzing and monitoring ML models in production. It provides a comprehensive set of tools for data and model validation, drift detection, and performance analysis. Some key features of Evidently include:

• 30+ built-in and customizable metrics for evaluating model performance, data quality, and data drift
• Automated reports and dashboards for visualizing model behavior over time
• Integration with popular ML libraries like scikit-learn, XGBoost, PyTorch, etc.
• Ability to run as part of a CI/CD pipeline for continuous monitoring
• Support for real-time and batch monitoring of both structured and unstructured data
• Alerting capabilities to notify relevant stakeholders when issues are detected

With Evidently, ML teams can gain deeper insights into their production models, proactively identify and troubleshoot issues, and maintain high performance and reliability. The library takes care of much of the heavy lifting involved in monitoring, allowing teams to focus on the insights instead of the infrastructure.

Key Metrics to Monitor with Evidently

Evidently provides a wide range of metrics out-of-the-box for monitoring different aspects of ML models and data. Here are some of the key metrics you should consider tracking:

Model Performance Metrics

• F1 score
• Precision
• Recall
• Accuracy
• ROC AUC
• Confusion matrix
• Prediction drift

These metrics give you a quantitative measure of how well your model is performing in production. By tracking them over time, you can identify when model performance starts to degrade and take proactive steps to retrain or update your model.

Data Quality Metrics

• Missing values
• Duplicates
• Data type mismatches
• Range violations
• Outliers
• Statistical metrics like min, max, mean, median, etc.

Poor data quality is one of the leading causes of model failure in production. Monitoring data quality metrics helps ensure your model is being fed high-quality, consistent data and can surface issues like data pipeline failures or schema changes.

Data Drift Metrics

• Numerical feature drift
• Categorical feature drift
• Label drift
• Prediction drift
• Dataset drift

Data drift refers to changes in the statistical properties of the input data compared to the data the model was trained on. Detecting data drift early is crucial as it can significantly degrade model performance over time. Evidently provides various statistical tests and visualizations to monitor drift at both the feature and dataset level.

In addition to these metrics, Evidently also allows you to define custom metrics tailored to your specific use case and business needs. It‘s important to choose the right metrics to monitor based on your model‘s purpose, the impact of failures, and the available data.

Monitoring in Action: Code Examples

Now that we‘ve covered the key concepts and metrics in ML monitoring, let‘s see how you can implement monitoring with Evidently in practice. We‘ll walk through a complete example of setting up monitoring for a machine learning pipeline.

Step 1: Install Evidently

First, make sure you have Evidently installed. You can install it using pip:

pip install evidently

Step 2: Prepare Your Data

Next, prepare your reference and production data for monitoring. The reference data is typically a holdout set from your model training process that represents the data your model was trained on. The production data is the actual live data your model is making predictions on.

For this example, we‘ll use a synthetic dataset for simplicity:

from evidently.datasets import get_iris_dataset

iris_reference = get_iris_dataset()
iris_production = get_iris_dataset(with_anomlies=True)  # add some anomalies to simulate drift

Step 3: Define Monitoring Metrics

Define the metrics you want to monitor using Evidently‘s TestSuite and TestPreset classes. Test suites allow you to group together related metrics and run them together. Test presets are predefined collections of metrics for common monitoring scenarios.

from evidently.test_suite import TestSuite
from evidently.test_preset import DataDriftTestPreset, DataQualityTestPreset

data_drift_suite = TestSuite(tests=[
    DataDriftTestPreset(),
])

data_quality_suite = TestSuite(tests=[
    DataQualityTestPreset(),
])

Here we‘ve defined two test suites: one for data drift and one for data quality, using the built-in presets. You can also define custom metrics using the individual metric classes like TestNumericFeatureDrift, TestDatasetMissingValues, etc.

Step 4: Run Monitoring Tests

Run the monitoring tests by calling the run method on the test suites with your reference and production data:

data_drift_suite.run(reference_data=iris_reference, current_data=iris_production)
data_quality_suite.run(reference_data=iris_reference, current_data=iris_production)

This will compute the defined metrics on your data and store the results in the test suite objects.

Step 5: Generate Reports

Generate human-readable reports of the monitoring results using Evidently‘s Report class:

from evidently.report import Report

report = Report(metrics=[
    data_drift_suite,
    data_quality_suite
])

report.save_html("monitoring_report.html")

This will create an HTML report with interactive visualizations of the monitoring metrics, making it easy to analyze and share the results.

Step 6: Set Up Alerting

To be notified when issues are detected, you can set up alerting based on the monitoring results. Evidently integrates with popular alerting solutions like Grafana, Prometheus, and Datadog.

For example, to set up a Grafana alert when data drift is detected:

from evidently.utils import create_grafana_alert

create_grafana_alert(
    test=data_drift_suite,
    url="http://my-grafana-instance.com/api/alerts",
    api_token="my-grafana-api-token",
    rule_name="Data Drift Alert",
    message="Data drift detected in production data",
    message_type="warning",
    rule_tags=["data_drift", "production"]
)

This will create a Grafana alert rule that triggers whenever the data drift suite detects drift above a certain threshold.

Integrating with MLOps Tools

Evidently can be easily integrated with other tools in your MLOps stack for a seamless monitoring workflow. Here are a few examples:

Prefect

Prefect is a popular open-source workflow orchestration tool for data science and engineering. You can use Prefect to schedule and run your Evidently monitoring pipelines as part of your overall ML workflow.

from prefect import task, Flow
from evidently.test_suite import TestSuite
from evidently.test_preset import DataDriftTestPreset

@task
def run_data_drift_tests():
    data_drift_suite = TestSuite(tests=[DataDriftTestPreset()])
    data_drift_suite.run(reference_data=reference_data, current_data=current_data)

    return data_drift_suite.as_dict()

with Flow("monitoring_flow") as flow:
    drift_results = run_data_drift_tests()

flow.run()

MLflow

MLflow is an open-source platform for managing the ML lifecycle, including experimentation, reproducibility, and deployment. You can log your Evidently monitoring results as MLflow artifacts to keep track of model performance over time.

import mlflow
from evidently.test_suite import TestSuite
from evidently.test_preset import DataQualityTestPreset

with mlflow.start_run():
    data_quality_suite = TestSuite(tests=[DataQualityTestPreset()])
    data_quality_suite.run(reference_data=reference_data, current_data=current_data)

    mlflow.log_artifacts(data_quality_suite.save("data_quality_results"))

Grafana

Grafana is a popular open-source platform for monitoring and observability. You can visualize your Evidently monitoring results in Grafana dashboards to get a real-time view of model performance.

Evidently provides built-in Grafana dashboards that you can import and customize:

from evidently.utils import export_grafana_dashboard

export_grafana_dashboard(
    test_suites=[data_drift_suite, data_quality_suite], 
    filepath="evidently_dashboard.json"
)

This will export a JSON file containing the dashboard definition, which you can then import into your Grafana instance.

Best Practices for Effective ML Monitoring

To get the most value out of ML monitoring with Evidently, here are some best practices to keep in mind:

1. Start with a baseline: Establish a baseline of your model‘s behavior on a reference dataset before deploying to production. This will make it easier to detect anomalies and drift later on.

2. Monitor multiple aspects: Don‘t just focus on model performance metrics. Monitor data quality, data drift, resource utilization, and other factors that can impact model health.

3. Set appropriate thresholds: Define acceptable thresholds for each metric based on your specific use case and business needs. This will help you avoid false alarms while still catching significant issues.

4. Use both batch and real-time monitoring: Batch monitoring is useful for catching long-term trends and drift, while real-time monitoring is essential for detecting acute issues and responding quickly.

5. Combine statistical and rule-based alerts: Statistical alerts based on deviation from a baseline are good for catching unexpected changes, while rule-based alerts can enforce specific business constraints.

6. Investigate and document issues: When an alert is triggered, thoroughly investigate the issue and document your findings. This will help you build up a knowledge base of common failure modes and how to resolve them.

7. Continuously update your monitoring: As your model and data evolve, make sure to update your monitoring setup accordingly. Regularly revisit your metrics, thresholds, and alerts to ensure they‘re still relevant and effective.

Conclusion

ML monitoring is a critical component of any production ML system. By proactively monitoring model performance, data quality, and drift, you can catch issues early and ensure your models are delivering value.

Evidently AI provides a powerful and flexible toolkit for implementing ML monitoring with minimal overhead. By following the examples and best practices outlined in this guide, you can set up comprehensive monitoring for your own models and pipelines.

As ML continues to mature and become more widely adopted, effective monitoring will only become more important. By staying on top of the latest tools and techniques, you can ensure your ML systems are reliable, performant, and driving real business impact. Happy monitoring!