MLOps: The Why and the What of Operationalizing Machine Learning at Scale

Machine learning (ML) has transformed from an academic niche to a major driver of business value across industries in recent years. IDC forecasts that worldwide spending on artificial intelligence (AI) and ML will reach $110 billion by 2024, with a compound annual growth rate of 20.1% [1]. However, organizations looking to harness the power of ML face significant challenges when it comes to deploying and maintaining ML models in production environments. This is where MLOps comes in.

MLOps, short for "machine learning operations", is an emerging practice for collaboration and communication between data scientists and IT operations professionals to help manage the production ML lifecycle. Applying DevOps practices to ML workloads, MLOps aims to increase automation and improve the quality of ML while also focusing on business and regulatory requirements.

The Need for MLOps

Deploying ML models into real-world production environments is no easy feat. ML systems are complex, consisting of much more than just the core model code developed by data scientists. As noted in a seminal paper, the actual ML code "is only a small fraction of the code in a ML system" once you factor in data collection, data verification, feature extraction, machine resource management, analysis tools, process management tools, serving infrastructure, and monitoring [2].

Without an MLOps framework in place to manage this complexity, ML projects often fail to deliver business value. Industry surveys have found that only 22% of companies using ML have successfully deployed a model into production [3]. Models that do get deployed are hampered by technical debt, suffer from performance degradation over time, lack reproducibility, and are difficult to update and maintain. The consequences are severe:

• 55% of organizations had an ML project fail due to issues with deployment [4]
• 87% of data science projects never make it to production [5]
• 50% of ML models initially deployed must be retrained or redeployed to maintain performance [6]

These challenges stem from a few root causes:

• Disconnect between data scientists and engineering/operations teams
• Lack of standardized tools, best practices, and frameworks for operationalizing ML
• Inherent differences between ML systems and traditional software

The goal of MLOps is to provide a structured approach to address these issues and streamline the entire ML lifecycle from experimentation to production and monitoring. By adopting MLOps, organizations can increase the agility, reliability, and scalability of deployed ML systems while better aligning development with business objectives.

MLOps vs Traditional DevOps

While MLOps shares many principles with DevOps for conventional software development, there are some key differences. The table below highlights some of the major ways MLOps differs from or extends standard DevOps:

Aspect	DevOps	MLOps
Code Assets	Application code	ML model + code
Testing	Unit & integration tests	Training/validation data tests
Versioning	Code versions	Data + model versions
Deployment	App/service deployment	Model serving
Monitoring	App performance	Model performance + data drift

MLOps must deal with the unique characteristics of ML workloads like managing data as a key artifact, model-data interaction, and monitoring for model-specific metrics. Later sections will dive deeper into these challenges.

Key Components of an MLOps Workflow

While specific MLOps tooling and implementations vary, a typical end-to-end MLOps workflow consists of the following components:

Use Case Discovery

The initial phase involves close collaboration between lines of business, data scientists, and other stakeholders to define the business problem and objectives an ML system will address. Key performance indicators (KPIs) and criteria for model evaluation and success are determined upfront.

Data Engineering

With objectives defined, the next step focuses on data. Data engineers work with data scientists to identify relevant data sources, acquire the data, and perform cleanup, integration, and preprocessing to get the data ready for modeling. Data validation is critical to catch issues like schema skew or training-serving skew that could negatively impact model performance down the line. Data lineage and provenance tracking are also key.

ML Pipeline and Experimentation

In this phase, data scientists iteratively develop models and evaluate them against KPIs established during use case discovery. An MLOps platform provides pipeline automation to streamline the process of training, tuning, and testing models. Experiments and associated artifacts are tracked to ensure reproducibility. The platform also enables continuous integration and delivery (CI/CD) so new models can be frequently deployed to production.

Production Deployment

Once a model is ready for production, the MLOps framework provides capabilities for secure, reliable deployment to the target environment whether that‘s a public cloud, on-premises data center, or an edge/mobile device. The model is packaged with its dependencies (e.g. as a container image) to ensure portability. Deployment is automated and may include canary rollouts, A/B testing, and rollbacks if issues arise.

Production Monitoring

Models running in production require continuous monitoring for data drift, concept drift, and degradation of performance against KPIs. The MLOps stack collects data and statistics to provide teams visibility into model behavior in the wild. Infrastructure monitoring is also needed to track memory, CPU, and other resource utilization. When issues or opportunities for improvement are uncovered, the MLOps workflow loops back with new experiments to develop updated models.

MLOps Challenges

MLOps aims to address a range of challenges teams face when operationalizing machine learning. Some of these overlap with classic DevOps issues while others are unique to ML. Key challenges include:

Collaboration and Alignment

MLOps must facilitate effective collaboration between data scientists, data engineers, IT operations, and business stakeholders. Without alignment, friction develops that can slow or derail ML initiatives. MLOps provides shared terminology, KPIs, and ways of working.

Automation and CI/CD

To enable rapid, frequent updates, MLOps automates key stages of the workflow like model building, testing, and deployment. Techniques from DevOps like CI/CD pipelines are adapted to the needs of ML (e.g. automating model validation in addition to code testing).

Reproducibility

Reproducing model behavior can be difficult when dealing with complex data dependencies, stochastic model training procedures, and changing environments. MLOps must systematically track data, code, model, and environment versions to ensure models can be reproduced later.

Data Management

Data quality is critical for building effective ML models, but data also tends to be the hardest asset to manage. MLOps platforms must deal with challenges like data drift, schema changes, and training-serving skew. Capturing data lineage is also key for auditing and compliance.

Model Deployment and Serving

Models must be packaged with their full set of dependencies to enable reliable deployment across environments. Containerization is commonly used to encapsulate models, but ML frameworks and models present unique requirements compared to traditional application code. Models also require specialized serving infrastructure for hosting and API access.

Monitoring and Observability

Deployed models must be continuously monitored to track online performance, data drift, model staleness, and resource utilization. This is critical for catching issues that can degrade model accuracy or cause unexpected behaviors in a changing real-world environment.

Governance and Compliance

ML models are increasingly subject to laws and regulations around data privacy, usage, bias, and explainability. MLOps must bake in capabilities for access controls, auditing, and interpretability features to comply with these evolving requirements.

Enabling New ML Use Cases

A robust MLOps foundation doesn‘t just help organizations better manage ML projects – it opens the door to new use cases entirely. Some emerging applications enabled by modern MLOps include:

• Real-time personalization: Rapidly training and deploying models to production empowers more real-time, granular personalization in customer-facing applications.

• Autonomous systems: MLOps provides the guardrails to deploy ML in high-stakes, production environments like industrial facilities, transportation networks, or smart cities.

• Enterprise-scale NLP: Deploying large language models for applications like semantic search or knowledge management requires the kind of complex serving infrastructure managed by MLOps platforms.

• Edge deployment: As more inference moves to the edge, MLOps enables deployment of ML on a massively distributed fleet of edge nodes and devices.

The most sophisticated, impactful applications of ML in coming years will be powered by strong MLOps foundations under the hood.

MLOps Trends and Predictions

As machine learning rapidly evolves from experimentation to production deployment, the MLOps ecosystem is advancing as well. A few key trends are shaping MLOps as we head into 2023 and beyond:

Accelerating Adoption

MLOps adoption is growing as more companies seek to deploy ML models into production. Over 80% of organizations now report MLOps as a priority initiative for their business [7]. We‘ve moved past the early-adopter phase, and a majority of enterprises investing in ML will likely have formal MLOps practices in place by 2025.

Maturing Platforms

Currently, the MLOps landscape is fragmented with a mix of open-source and commercial point solutions. As the market matures, there will likely be consolidation to full-lifecycle platforms that can manage ML end-to-end. By 2024, 60% of ML platforms will include built-in MLOps capabilities [8].

Convergence with Adjacent Disciplines

Expect tighter integration across MLOps, DataOps (managing the data pipeline feeding ML), AIOps (AI for IT operations), and overall DevOps. Over 70% of companies adopting MLOps plan to also invest in DataOps [9]. Forward-looking organizations will blend these practices for a holistic approach to operationalizing data science and ML.

New Standards and Interoperability

While still a relatively nascent field, common standards and best practices will solidify around MLOps. This includes areas like metadata and API formats, testing methodologies, and monitoring approaches. Open-source projects like MLflow and KubeFlow are already helping drive interoperability.

Talent and Upskilling

As MLOps grows, so does the demand for engineering talent with the right skills. Over 90% of enterprises cite lack of ML skills as a top challenge for adoption [10]. Expect investment in training, certifications, university partnerships, and upskilling programs to build the MLOps workforce.

Conclusion

Machine learning offers immense value for organizations that successfully deploy ML to production, but the path is littered with challenges. MLOps provides a framework to navigate these obstacles, leveraging proven practices from DevOps while also tackling the unique characteristics of ML systems.

By adopting MLOps best practices, organizations can improve the velocity, reliability, and impact of ML initiatives. Teams can deliver more models to production with fewer failures, respond faster to changing data and business requirements, and ultimately achieve better outcomes from their ML investments.

As Gartner notes: "MLOps will be a necessary element of data science and ML projects that move toward production. Without MLOps, ML models have problems in deployment, scaling and, most importantly, quality" [11]. The future of ML is bright – but only with a solid MLOps foundation.

References

[1] IDC, "Worldwide Artificial Intelligence Spending Guide", https://www.idc.com/tracker/showproductinfo.jsp?prod_id=1732

[2] D. Sculley et al., "Hidden Technical Debt in Machine Learning Systems", NIPS 2015.

[3] Algorithmia, "2021 Enterprise Trends in Machine Learning", https://info.algorithmia.com/hubfs/2020/Reports/2021-Trends-in-ML/Algorithmia_2021_Enterprise_ML_Trends.pdf

[4] Dimensional Research, "Artificial Intelligence and Machine Learning Projects Obstructed by Data Issues", https://dimensionalresearch.com/ai-machine-learning-projects-obstructed-by-data-issues/

[5] VentureBeat, "Why do 87% of data science projects never make it into production?", https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/

[6] Accenture, "AI: Built to Scale", https://www.accenture.com/_acnmedia/Thought-Leadership-Assets/PDF-2/Accenture-Built-to-Scale-PDF-Report.pdf

[7] Deloitte, "State of AI in the Enterprise", https://www2.deloitte.com/us/en/insights/focus/cognitive-technologies/state-of-ai-and-intelligent-automation-in-business-survey.html

[8] Gartner, "Innovation Insight for MLOps", https://www.gartner.com/doc/3978994/innovation-insight-mlops

[9] 451 Research, "2022 Trends in DataOps", https://go.451research.com/2022-Trends-DataOps-ML-Strategists.html

[10] O‘Reilly, "The State of Machine Learning Adoption in the Enterprise", https://www.oreilly.com/radar/the-state-of-machine-learning-adoption-in-the-enterprise-2020/

[11] Gartner, "How to Overcome the Top Challenges to Adopting MLOps", https://www.gartner.com/document/4002580