Functions, Users, and Comparative Analysis
We decided that Docs should have prime location.
Build AI products you can trust.
We’re super excited to share that Aporia is now the first ML observability offering integration to the Databricks Lakehouse Platform. This partnership means that you can now effortlessly automate your data pipelines, monitor, visualize, and explain your ML models in production. Aporia and Databricks: A Match Made in Data Heaven One key benefit of this […]
Fundamentals of ML observability
Metrics, feature importance and more
We’re excited 😁 to share that Forbes has named Aporia a Next Billion-Dollar Company. This recognition comes on the heels of our recent $25 million Series A funding and is a huge testament that Aporia’s mission and the need for trust in AI are more relevant than ever. We are very proud to be listed […]
Introduction and agenda >
Machine learning (ML) models can provide valuable insights, but to be effective, they need to continuously access and efficiently analyze an organization’s data assets. Machine Learning Operations (MLOps) is a set of tools, methodologies, and processes that enable organizations to build and run ML models efficiently.
MLOps is a cross-functional, iterative process that helps organizations build and operate data science systems. It lends from DevOps practices, treating machine learning (ML) models as reusable software artifacts. This allows models to be deployed and continuously monitored in a repeatable process.
MLOps supports continuous integration (CI), and rapid, automated deployment for ML models. To address the problem of model drift and data drift, it performs continuous monitoring and retraining of models, based on performance metrics in production environments, to ensure they perform optimally as data and context change over time.
Here are the main issues and bottlenecks facing ML projects, which MLOps practices can help address:
DevOps and MLOps have many similarities, because the MLOps process was derived from DevOps principles. But there are a few key differences:
While commonly confused, MLOps and AIOps are two distinct fields:
The problem solved by AIOps is that organizations are generating huge volumes of operational data, and it is increasingly difficult to identify risks and alert staff to resolve them. AIOps technology can identify issues, and automatically resolve recurring issues, without requiring staff to manually monitor processes.
AIOps combines big data and machine learning to automate IT operational processes such as event correlation, anomaly detection, and causality determination. This can provide insights and predictive analytics to help IT operations effectively respond to operational problems.
Let’s review the basic building blocks and workflow of an MLOps process. These are illustrated in the diagram below.
The process works as follows:
In a full MLOps pipeline, all steps in the process are automatic, but can be optionally stopped by operators at any time for manual evaluation, or extended with specific steps required by the organization. The pipeline can be activated on several triggers—when new data is available for retraining, when the model is updated, or when performance issues are discovered in a production model.
This discussion is based on the MLOps maturity model published by Google Cloud.
In a manual MLOps process, every step is manual, from initial data analysis through preparation, training, and validation. There is a disconnect between ML and operations teams—data scientists build a model and hand it off to operations teams who must figure out how to deploy it.
Because of the difficulty of creating and deploying new versions of a model, releases happen infrequently, usually only 2-4 times per year.
In this manual process, there is no continuous integration (CI) system, meaning that model code is written in notebook systems and either shared as files or committed to source control. There is also no continuous deployment (CD), meaning that deployment is performed manually. Model deployment is only concerned with the prediction service (typically a REST API), not the entire MLOps system, and there is no active production monitoring.
At this level of maturity, the following improvements are added:
At this level of maturity, the following improvements are added:
MLOps can be hosted on-premises and in the cloud, and each has its own advantages:
All three major cloud providers offer MLOps platforms that can help organizations of all sizes manage an ML pipeline in the cloud:
Related content: Read our guide to Azure MLOps (coming soon)
One of the biggest barriers to implementing MLOps is the lack of computing power. Machine learning algorithms require a lot of resources to run. On-premises systems often struggle to adequately meet these compute requirements for large-scale ML projects.
Another issue is that the MLOps process requires training models multiple times for automated training, testing and evaluation of every model iteration—this increases computational requirements by an order of magnitude.
A natural solution for this problem is the computing power provided by cloud platforms. Cloud providers offer elastically scalable resources, which can automatically provision enough computing power to perform all tasks required by an ML project, from data preparation to model training to model inference. Most MLOps programs rely on the use of cloud-native solutions.
Machine learning algorithms require large amounts of data to obtain high-quality results. However, many organizations retrieve input data for ML algorithms from siloed data stored in different locations and formats.
To make data usable for ML, organizations need a data platform that can ingest data in multiple formats, both structured and unstructured, store it in a central repository, pre-process and normalize it to enable consistent analysis, and apply data security and governance to protect sensitive data.
Related content: Read our guide to AI governance (coming soon)
ML projects must integrate multiple technologies, including data processing, machine learning and deep learning frameworks, CI/CD automation tools, monitoring and observability tools, and more. Creating a cohesive MLOps pipeline is a challenge.
Several cloud providers offer an all-in-one MLOps platform that deals with everything from data ingestion through to final model deployment. This can solve the integration challenge, but it also requires locking into a specific cloud vendor, and might be difficult to customize to the organization’s specific requirements.
Simply put, more advanced automation increases an organization’s MLOps maturity and will probably lead to better results.
In an environment without MLOps, much of the work of machine learning systems is done manually. These tasks include cleaning and transforming data, engineering features, partitioning training, testing data, writing model training code, and more. This manual effort leaves room for error and wastes the valuable time of data science teams.
One example of automation that can reduce manual labor is retraining—an MLOps pipeline can automatically perform data collection, validation of the model on the new data, experimentation, feature engineering, model testing and evaluation, with no human intervention. Continuous retraining is considered one of the first steps in automating machine learning.
By automating more and more stages of the ML workflow, an organization can eventually reach a fully streamlined ML development process that enables rapid iteration and feedback, in line with agile and DevOps principles.
Experiments are a crucial part of the ML process. Data scientists experiment with data sets, features, machine learning models, and hyperparameters. In this process, it is important to track each iteration of the experiment to finding the best combination of criteria that can improve model performance.
Traditionally, data scientists ran experiments using notebook platforms, often running on their local machine, manually tracking model parameters and details. They would often need to wait for models to train, due to limited computing resources, and there was no central way to log and share experiment results, leading to errors, inconsistencies, and duplicate work.
While Git can be used to perform version control for model code, it cannot easily be used to log the results of experiments data scientists perform. This requires the concept of a model registry—a central repository of ML models which can track performance and other changes across multiple ML models and many different variations of the same model.
Rapid experimentation, with consistent tracking of experiments, allows MLOps teams to identify successful models, roll back models that are not performing well, make results reproducible, and provide a complete audit trail of the experimentation process. This significantly reduces manual work for data scientists, freeing up more time for real experimentation.
To achieve a mature MLOps pipeline, organizations must evolve. This requires process changes that encourage collaboration between teams, breaking down silos. In some cases, the entire team needs to be restructured to promote MLOps principles.
In less mature data science environments, data scientists, engineers, and software engineers often work independently. As maturity increases, all members of the team must work as a cohesive unit. Data scientists and engineers must work together to turn experimental code into repeatable pipelines, and software and data engineers must work together to automatically integrate models into application code.
More collaboration means less reliance on any one person throughout the deployment. Teamwork, combined with automated tooling, can help reduce expensive manual work. Collaboration is key to achieving the level of automation required for a mature MLOps program.
Aporia is a full-stack, customizable machine learning observability platform that empowers data science and ML teams to trust their AI and act on Responsible AI principles. When a machine learning model starts interacting with the real world, making real predictions for real people and businesses, there are various triggers – like drift and performance and model degradation – that can send your model spiraling out of control.
Our ML observability platform is the ideal partner for Data Scientists and ML engineers to visualize, monitor, explain, and improve ML models in production in minutes. The platform supports any use case and fits naturally into your existing ML stack alongside your favorite MLOps tools. We empower organizations with key features and tools to ensure high model performance:
Root Cause Investigation
To get a hands-on feel for Aporia’s ML monitoring solution, we recommend:
Authored by Aporia
What is Machine Learning Engineering?
Authored by Datagen
Authored by Cynet