Streamlining Machine Learning Workflows with MLOps

Machine learning (ML) is becoming a core driver of value for digitally-native companies as well as traditional enterprises undergoing digital transformation. IDC forecasts that worldwide spending on artificial intelligence (AI) will top $110 billion by 2024, with a compound annual growth rate of 20.1% [1]. However, the reality is that many organizations still struggle to translate ML experiments into production deployments that drive real business impact.

A 2020 NewVantage Partners survey found that only 11% of firms have deployed AI capabilities into widespread production [2]. Why is there such a large gap between ML experimentation and operationalization? A key reason is that developing and deploying ML systems introduces new complexities and failure modes compared to traditional software.

This is where MLOps comes in. MLOps, short for "Machine Learning Operations," is an emerging practice that aims to systematize and streamline the end-to-end process of developing, deploying and maintaining ML models in production.

Why MLOps Matters

A typical ML workflow involves numerous steps, each with its own challenges:

Data preparation – Acquiring, cleansing, labeling, and securely storing large volumes of data in a reproducible manner
Model development – Iterative process of feature engineering, model training, hyperparameter tuning, and evaluation
Model deployment – Packaging models and their dependencies for deployment on heterogeneous infrastructure, often requiring close collaboration between data and ops teams
Model monitoring – Detecting model performance degradation, data drift and resource bottlenecks in production
Model lifecycle management – Managing multiple deployed model versions, handling rollbacks, A/B testing and progressive delivery

Ad-hoc artisanal approaches to these challenges result in friction, failures, and ultimately low ML velocity and value. In a 2019 survey, Algorithmia found that 55% of companies take between 7-18 months to deploy a single ML model [3].

MLOps addresses these challenges by applying DevOps principles of continuous integration (CI), continuous delivery (CD), observability and automation to the unique needs of ML systems. Key goals of MLOps include:

Enabling rapid experimentation and iteration between data scientists and engineers
Automating the ML pipeline from data to deployment to monitoring
Increasing reproducibility and traceability of model training and predictions
Providing observability into model performance and resource usage
Supporting model testing, validation and compliance processes
Enabling frequent delivery of updated models to production safely

When implemented well, MLOps can significantly improve the quality, velocity and reliability of ML initiatives. Google‘s MLOps transformation increased the number of ML models deployed to production from a few hundred to tens of thousands, while reducing average training-to-deployment time from months to weeks [4].

Key Concepts and Components of MLOps

Figure 1: Key components of an end-to-end MLOps pipeline (Source: GigaOm)

Let‘s dive deeper into some of the key aspects of MLOps:

Data Management

Reliable, high-quality data is the foundation of successful ML systems. MLOps employs tools and practices to enable:

Data versioning and lineage tracking with tools like DVC, Pachyderm and Delta Lake
Automated data pipelines for extraction, validation and feature engineering using Airflow, Kubeflow Pipelines or AWS Step Functions
Feature stores to serve curated features to models in training and production, e.g. Tecton, Feast
Data quality monitoring to detect schema drift and anomalies, e.g. Monte Carlo, Databand

A robust data management layer is crucial to support rapid experimentation and ensure models are trained on consistent, reliable data.

Experimentation

The key to developing high-quality ML models is enabling rapid iteration and experimentation by data scientists. MLOps supports this through:

Notebook environments like Jupyter, Databricks and AWS SageMaker Studio configured with access to data, compute and essential libraries
Experiment tracking tools like MLflow, Weights & Biases, and Comet.ml to log model hyperparameters, code, dataset versioning and evaluation metrics
Distributed training infrastructure using Kubernetes, Kubeflow, SageMaker or Ray to parallelize model training on GPUs and TPUs
Automated hyperparameter tuning with techniques like random search, Bayesian optimization and genetic algorithms
Reproducible experimentation with containerized environments and immutable data snapshots

Giving data scientists self-service access to infrastructure with strong experiment tracking is key to increasing model velocity while maintaining reproducibility.

Model Deployment

Deploying ML models into production with reliability and observability requires close collaboration between data science and engineering teams. Key components include:

Automated CI/CD pipelines that build, test, and deploy model prediction services triggered by code or data changes
Model registries like MLflow Registry to track model versions, metadata and deployment history
Model serving infrastructure leveraging containers and orchestration platforms like Kubernetes, SageMaker, or KFServing
Edge deployment on mobile and IoT devices using optimized model formats like TensorFlow Lite, ONNX, and CoreML
A/B testing and progressive rollout techniques to update models without service disruption
Security practices like encryption of data and model artifacts, access controls, and audit logging

Applying DevOps best practices to model deployment is essential for safe and frequent releases of ML applications.

Monitoring

Deployed ML models can degrade in ways that are difficult to detect without proactive monitoring. Key signals to monitor include:

Model prediction performance (e.g. accuracy, precision, recall, F1 score)
Model prediction distribution, bias and fairness metrics
Input data quality, schema conformance and drift from training data
Latency and throughput of model prediction services
Resource utilization (e.g. CPU, GPU, memory usage)

Tools like WhyLabs, Arthur, Fiddler and Evidently can help instrument and analyze models to detect regressions and anomalies. When regressions are detected, auto-rollback or retraining may be triggered.

Infrastructure as Code

Managing the complex, distributed infrastructure required for ML workloads is a key pain point. Applying infrastructure-as-code (IaC) practices can help make environments reproducible and testable. Key approaches include:

Declarative templates using Terraform, CloudFormation, and Deployment Manager to specify infrastructure and configuration
Kubernetes-native abstractions like Kubeflow that define ML workflows as declarative pipeline resources
Spark-native approachs like Databricks MLFlow that automate cluster management and distribution of ML workoads
End-to-end data science platforms like Sagemaker, Dataiku, and Domino Data Lab that provide integrating tooling for the full lifecycle

IaC allows ML teams to rapidly provision and scale infrastructure while minimizing configuration drift and failures.

End-to-End MLOps Workflow Example

Let‘s walk through an example of what an end-to-end model development and deployment process might look like using MLOps principles:

Data scientist develops a new model in a Jupyter notebook environment, experimenting with different feature sets and model architectures. Experiment parameters, evaluation metrics and models are logged with MLflow:
```
import mlflow 

with mlflow.start_run():
    mlflow.log_param("num_trees", 200)
    mlflow.log_param("max_depth", 10)
    mlflow.sklearn.log_model(model, "model")
    mlflow.log_metric("accuracy", 0.92)
```
Data scientist checks the model code into a Git repository, which triggers an automated CI/CD pipeline. The pipeline trains the model on a full dataset, runs unit and integration tests, and registers the trained model in the MLflow registry.

If tests pass, the model is packaged with its dependencies using a tool like BentoML and deployed to a staging environment for further validation. The staging environment is provisioned using a Terraform template:

resource "aws_sagemaker_model" "example" {
  name               = "example-model"
  execution_role_arn = aws_iam_role.example.arn

  primary_container {
    image = "012345678901.dkr.ecr.us-west-2.amazonaws.com/example-model:1.0"
    model_data_url = "s3://example/model.tar.gz"
  }
}

After manual review, the model is promoted to production using a Kubernetes Deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-model
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-model
  template:
    metadata:
      labels:
        app: example-model
    spec:
      containers:
      - name: example-model
        image: 012345678901.dkr.ecr.us-west-2.amazonaws.com/example-model:1.0
        ports:
        - containerPort: 8080

In production, the model‘s predictions and input data distributions are monitored using WhyLabs. When data drift is detected, the pipeline is triggered to retrain and deploy an updated model.

Of course, every organization‘s MLOps stack and workflow will differ based on unique tools, skills and needs. However the general principles of automation, reproducibility, observability and agility apply universally.

Organizational Maturity Model for MLOps

Figure 2: MLOps Maturity Model (Source: ml-ops.org)

Adopting MLOps practices is a journey that requires changes to people, processes and technology. The maturity of an organization‘s MLOps capabilities can be assessed along several dimensions:

Release velocity – Manaul, monthly release to on-demand release of ML models
Automation – From manual scripts to fully automated data and ML pipelines
Testing – From ad-hoc to automated model evaluation, validation and A/B testing
Monitoring – From no monitoring to auto-detection of model drift and data quality issues
Explainability – From black box models to documentation of features, models and decisions

The goal of MLOps should be to help organizations advance along these dimensions to increase agility, reliability and transparency of deployed ML systems. Ongoing measurement and optimization is required for progress.

Getting Started with MLOps

Putting MLOps into practice requires close collaboration between data scientists, data engineers, ML engineers and operations teams. Some key steps to get started:

Align teams on the goals and success metrics of your ML initiative. Foster shared terminology and incentives.
Map out your current ML workflows and identify bottlenecks and opportunities for automation. Focus on the highest value improvements first.
Establish a common ML stack aligned with existing skills. Adopt managed services where possible to limit undifferentiated heavy lifting.
Start by automating a single model use case end-to-end from data prep to serving to monitoring. Measure KPIs like cycle time and model performance.
Incrementally expand the scope of automation across more models and teams. Continuously gather feedback and optimize processes.
Invest in education and evangelism of MLOps practices. Encourage teams to stay current with the rapidly evolving ecosystem of tools and techniques.

The most important thing is to get started and aim for continuous, iterative improvement over time. Don‘t try to boil the ocean from the outset.

Conclusion

MLOps is still a nascent field but is rapidly evolving to help organizations overcome the hurdles to successful operationalization of machine learning. By adopting MLOps practices, data science teams can improve the velocity, quality and impact of their ML initiatives.

As the ML ecosystem continues to mature, expect to see more convergence and standardization around best practices and tooling. However, the core principles of automation, reproducibility, observability and continuous improvement will remain central to MLOps.

Ultimately, MLOps is about creating a culture and environment where data scientists and engineers can collaborate effectively to solve high-value problems with ML. Organizations that embrace this mindset will be well-positioned to unlock the transformative potential of AI.

References

IDC Forecasts Improved Growth for Global AI Market in 2021, IDC
AI adoption in the enterprise 2020, O‘Reilly
2020 State of Enterprise Machine Learning, Algorithmia
Introducing the AI Adoption Framework, Google Cloud Blog

Streamlining Machine Learning Workflows with MLOps

Why MLOps Matters