Everything you need for AI Performance in one platform.
We decided that Docs should have prime location.
Fundamentals of ML observability
Metrics, feature importance and more
We’re excited ???? to share that Forbes has named Aporia a Next Billion-Dollar Company. This recognition comes on the heels of our recent $25 million Series A funding and is a huge testament that Aporia’s mission and the need for trust in AI are more relevant than ever. We are very proud to be listed […]
Introduction and agenda >
Machine learning (ML) is a branch of artificial intelligence (AI) in which machines learn from data and past experience to recognize patterns, make predictions, and perform cognitive tasks, without being explicitly programmed. Machine learning models can learn and adapt to new patterns by training on datasets that provide relevant examples.
Machine learning brings computer science and statistics together to create novel predictive models. Machine learning systems learn through an iterative process, using their training data to build a mathematical model that can make predictions on the data. There are thousands of machine learning algorithms available.
Data scientists aim to select the most appropriate algorithm for their problem, train it on a high quality dataset, and tune its hyperparameters to achieve the best performance. However, training an ML model is not a one-time event. After a model is deployed to production and is used for inference (providing responses to real world queries), it is essential to monitor its performance and continue improving it with new data and ongoing tuning.
According to Gartner, supervised learning is currently the most widely used type of machine learning among enterprises. This type of machine learning provides labeled data containing historical inputs and outputs to a machine learning algorithm, and the data is transformed into a model that can produce similar outputs for new, unseen data.
Common algorithms: deep neural networks, decision trees, linear regression, and support vector machines (SVM).
Use cases include: data classification, financial forecasting, fraud detection.
While supervised learning requires data provided by its operators, unsupervised learning does not require a labeled training set and data. Instead, it tries to identify patterns directly in production data. This type of machine learning is useful when you need to identify patterns and make decisions using data, and historical data is not available.
Common algorithms: hidden Markov models, k-means, hierarchical clustering, and Gaussian mixture models.
Use cases include: customer segmentation, recommender systems, data visualization.
Semi-supervised learning is a machine learning algorithm that combines supervised and unsupervised learning algorithms. During training, it uses a combination of labeled and unlabeled data sets.
A disadvantage of supervised learning is that it requires expensive manual data labeling. On the other hand, unsupervised learning is limited in scope. To overcome these shortcomings, the concept of semi-supervised learning combines these two paradigms to provide models that can work with only a limited number of labeled data samples and still provide powerful capabilities.
Reinforcement learning is a feedback-based process. The algorithm defines the actions of an agent, which can take action, learn from experience, and improve performance in an experimental way. The agent is rewarded for performing the correct step and penalized for performing the wrong step. Reinforcement learning agents aim to maximize rewards by taking the most appropriate actions.
Unlike supervised learning, in reinforcement learning there is no labeled data and the agent learns only from experience. For example, agents can learn via a game, in which they take actions and receive feedback through penalties and rewards that affect overall game score. The goal of the agent is to get a high score.
Common algorithms: SARSA-Lambda, DQN, DDPG, Actor-Critic
Use cases include: game theory, simulating synthetic environments, multi-agent systems
Deep learning is a branch of machine learning that uses layered algorithms to better understand complex data. Unlike previous generations of machine learning technology, such as regression models, deep learning algorithms are not limited to generating interpretable sets of relationships. Instead, deep learning relies on layers of non-linear connections to generate interactive, distributed representations based on thousands or even millions of factors.
Given a large training dataset, a deep learning algorithm can identify relationships between virtually any elements. These relationships can exist between shapes, colors, text, or any other input. When properly trained and tuned, the system can be used to generate predictions that approach the cognitive abilities of humans.
Common algorithms: multilayer perceptron (classic artificial neural network), convolutional neural network (CNN), recurrent neural network (RNN)
Use cases include: computer vision, machine translation, conversational AI
Learn more in the detailed guide to machine learning models.
Automatic speech recognition (ASR), also known as computer speech recognition or speech-to-text, is the ability to use natural language processing (NLP) to convert human speech into written form. For example, many mobile devices have voice recognition built into the system to perform voice searches.
This artificial intelligence technology allows computers to derive meaningful information from digital images, videos, and other visual inputs and take appropriate action. Computer vision with convolutional neural networks (CNN) has applications such as photo tagging in social media, medical radiography, and autonomous vehicles.
Learn more in the detailed guides to:
Face recognition uses machine learning algorithms to determine the similarity of two facial images, to evaluate a claim to identity. This technology is used for everything from logging a user into a mobile phone to searching a database of photos for a specific person.
Facial recognition converts facial images into digital representations, which are processed by neural networks to obtain high quality features called face embeddings. These embeddings are compared to determine similarity.
Learn more in the detailed guide to face recognition
With the proliferation of rich media on websites and social networks, image and video editing is becoming more common among organizations and individuals around the world. Traditionally, these were time-consuming manual tasks, but many image and video editing tasks can now be performed by AI algorithms that surpass humans.
AI algorithms analyze photos and make intelligent predictions about how to edit, adjust, and enhance them. This eliminates manual labor and saves time and money for content creators. For large media organizations, this can result in significant cost savings and a more flexible content creation process.
With the help of artificial intelligence, organizations can also create more personalized videos to increase engagement. AI-powered video applications provide end-users with powerful features such as video search for important moments, and the ability to automatically create professional-looking video clips in a few clicks.
Learn more in the detailed guides to:
AI algorithms analyze historical behavior data to identify data trends that can be used to develop more effective cross-sell strategies. Online retailers use this method to recommend related products to customers.
Learn more in the detailed guide to recommender systems
Fraud detection involves identifying commercial or financial transactions that have illegal or malicious intent. Traditionally, fraud detection systems were based on static rule-based systems, which were maintained by expert human analysts. They were difficult to maintain and could miss new types of fraud that existing rules did not capture.
Modern fraud detection systems are based on machine learning algorithms, which detect special features in fraudulent transactions that legitimate transactions do not have. ML models can detect suspicious patterns in transactions, calculate a probability that the transaction is fraudulent, and if it passes a certain threshold, flag it for human investigation.
Banks and other financial institutions can use machine learning to find suspicious transactions. Supervised learning allows you to train a model using information about known fraudulent transactions. Anomaly detection identifies transactions that are unusual and require further investigation.
Advanced Threat Protection (ATP) is a set of practices and solutions that can be used to detect and prevent advanced malware and attacks.
Advanced threat protection solutions leverage User and Entity Behavior Analysis (UEBA), based on machine learning algorithms, to reduce false positives and identify real security incidents.
Learn more in the detailed guide to advanced threat protection
Fuzzing is a technique for automatically detecting errors. The purpose of fuzzing is to overload an application, causing unexpected behavior, resource leaks, or crashes.
This process uses invalid, unexpected data, or random data as input to a computer system. The fuzzer repeats this process, monitoring the environment until it detects a vulnerability. Fuzzing often leverages machine learning to create new and unexpected inputs that could help uncover weaknesses in the application.
Learn more in the detailed guide to fuzzing
Body segmentation is a lesser-known but equally critical application of machine learning. It involves the segmentation of human bodies in images or videos, which can have various applications, from gaming and virtual reality to healthcare and fitness.
Machine learning algorithms are trained to identify and segment different parts of the human body, understanding the human form in more detail than ever before. This can enable the creation of more immersive gaming experiences, more accurate fitness tracking, and even improved medical diagnostics. Imagine a game that can track your body movements in real-time, a fitness app that can analyze your posture, or a medical imaging system that can identify anomalies in body structures.
Learn more in the detailed guide to body segmentation
One of the most significant trends in machine learning is the move towards cloud-based solutions. Cloud platforms offer a host of benefits for machine learning, including scalability, flexibility, and cost-effectiveness. They allow businesses to quickly scale up their machine learning efforts without the need for significant upfront investment.
Moreover, cloud platforms provide access to cutting-edge machine learning tools and frameworks, enabling businesses to tap into the latest advancements in the field. They also offer a collaborative environment where data scientists, developers, and business stakeholders can work together to develop and deploy machine learning models.
Learn more in the detailed guide to machine learning in the cloud
Feature importance is a measure that indicates how much each feature in a machine learning model contributes to its overall predictive power. It helps in understanding which features are more relevant in making predictions and can be used for model interpretability.
There are different methods to compute feature importance, such as permutation importance, mean decrease impurity, and coefficient magnitudes. These methods assign a score or weight to each feature, indicating its relative importance in the model.
Model interpretability is the process of understanding how a model makes its predictions or decisions. By examining the feature importance scores, data scientists can identify which features have the most significant impact on the model’s predictions. This information can be used to explain how the model works to stakeholders and to gain insight into the underlying patterns in the data.
Furthermore, feature importance can be used to identify and remove redundant or irrelevant features from the model. This can simplify the model and make it more interpretable, while also improving its performance by reducing the risk of overfitting.
Learn more in the detailed guide to feature importance.
MLOps stands for Machine Learning operations. It is a key function in machine learning engineering, focused on simplifying the process of deploying, maintaining and monitoring machine learning models in production. MLOps is often a collaborative function of data scientists, DevOps engineers, and IT operations.
MLOps is a way to help create and improve the quality of machine learning and AI solutions. By adopting the MLOps approach, data scientists and machine learning engineers can work together to implement continuous integration and deployment (CI/CD) practices and appropriate monitoring, validation, and governance of ML models. The end goal is to accelerate model development and production, while improving model performance and quality.
Learn more in the detailed guide to machine learning operations (MLOps)
ML monitoring is a set of techniques for observing the performance of ML models in production. ML models are typically trained by observing an example dataset, and minimizing errors that indicate how well the model performs on the training task.
Once deployed to production, ML models apply the learnings from their training data to new, real-world. However, many factors, including differences between the initial training data and real-world production data, can degrade production model performance over time.
An effective machine learning monitoring system can detect these changes and help data science teams continuously improve models and datasets. In the absence of monitoring, a model can fail silently, which can have a serious negative impact on business performance and end-user experience.
Explainable Artificial Intelligence (XAI) is a set of processes and methods that enable human stakeholders to understand and trust the outputs of machine learning algorithms.
Explainable AI is used to describe AI models and explain their decisions, expected impacts, and potential biases. It helps characterize model accuracy, fairness, transparency, and outcomes in AI-powered decision-making.
Explainable AI is critical to building trust and confidence when organizations deploy AI models into production. AI explainability also helps organizations adopt a responsible approach to AI development.
AI needs a lot of data to produce good results. Synthetic data is an important source of large data sets, which can help model phenomena where data is difficult to obtain, or to capture edge cases that don’t occur often in real life.
Synthetic data is artificially generated through machine learning algorithms. It reflects the statistical nature of real-world data, but does not use any identifying characteristics (such as names or personal information). Therefore, it reduces the privacy and compliance risks raised by AI datasets.
Learn more in the detailed guide to synthetic data
A machine learning model is a program trained to recognize certain types of patterns in order to perform a useful cognitive task (for example, see the Machine Learning Use Cases above). It contains algorithms that can be trained based on a dataset, and can then learn from that data, and apply it to make predictions on new unseen data.
Developing machine learning models is a new activity for many organizations. It is a complex process that requires diligence, experimentation, and creativity. Below we describe the key steps involved in the process.
There are thousands of machine learning algorithms, and it can be difficult to determine the best algorithm for a given model. In most cases, you will try multiple algorithms to find one that provides the most accurate results.
Key considerations for selecting an algorithm include the size of the training data, the required accuracy and interpretability of model outputs, the training speed required, linearity of the training data, and number of features in the data set.
By splitting the training data into two or more groups, you can train and validate the model using a single data source. This allows you to determine if the model is overfitting—meaning that it works well on the training data, but not on the unseen test data.
Most machine learning projects divide the dataset into three groups:
What is cross validation?
Cross validation is a common technique to partition training data, in order to maximize its value for model training. For example, 10-fold cross-validation splits the data into 10 groups, so you can train and test the data 10 times. This works as follows:
The average performance of the model across all 10 tests is called the cross-validation score.
Note that in some types of data, such as time series data sets, cross validation works differently than described above.
Hyperparameters are model properties that data science teams set before building and training models. They are external parameters that determine how the model operates, and are treated separately from model parameters, which are dynamically determined as the model trains.
For example, in a neural network, there are several hyperparameters including the number of neural layers and the learning rate. Data scientists set these hyperparameters and then train the neural network to get the model parameters, which are the weights and biases.
It is common to re-run the model with multiple combinations of hyperparameters to see which combination provides the best results.
Once a data science team has selected an algorithm, the data is ready, and model hyperparameters have been determined, it’s time to start training the model. This process cycles through each set of hyperparameter values you decide to investigate. Data points are typically fed to the model in groups known as batches. The model may process the data in one or more cycles, known as epochs – each including one or more batches. After each epoch, cross validation is performed to see the model’s performance.
At this stage It is common to test multiple algorithms, each with multiple hyperparameter variations, to see which provides the best results.
Earlier in the process, the dataset was divided into three groups—training set, validation set, and testing set. Now that a data science team has obtained a final version of the model, they can subject it to a realistic performance test using the testing set, which the model has not seen yet.
Applying the final version of the model to the testing dataset, and measuring performance metrics, simulates how the model will perform on real-world data. The team can compare performance to other, state of the art models, or other experiments conducted by themselves or their colleagues.
If a model’s performance is insufficient, the team can go back to the drawing board and try to build a better model, by changing the algorithm, hyperparameters, or improving the dataset.
Learn more in the detailed guide to data training
Machine learning engineering is the use of engineering principles, tools, and techniques to design and build complex machine learning systems. Machine learning engineers are responsible for data collection, model training, building and deploying a machine learning system that customers can use. They enable machine learning algorithms to be implemented as part of efficient production systems.
Difference between data scientists and data engineers
A dataset is a curated collection of data for a machine learning project. Computer vision models make use of image datasets, which contain curated digital images used to test, train, and evaluate the performance of image processing algorithms. Image datasets help algorithms learn how to recognize information in images and perform relevant cognitive activities.
Image annotations are a way to label an image or set of images. Operators or domain experts view a series of images, identify related objects in each image, and annotate the images for the relevant task, such as classification or object segmentation. This typically involves marking each object’s shape and label. These annotations can then be used to generate training data sets for computer vision models.
Models take the annotated images as input, using human annotations as their “ground truth”. Based on this ground truth, they learn to independently detect objects and label images. This process can be used to train models for tasks such as image classification, object recognition, and image segmentation.
The longest and most resource-intensive phase of most deep learning projects is the training phase. For a model with a large number of parameters, training time can be significant. When training takes longer, and insufficient computing power is available, teams wait and waste valuable time. This also makes it difficult to experiment with multiple variations of algorithms and hyperparameters.
Traditional central processing units (CPUs) can be slow to process machine learning computations; graphics processing units (GPUs) can accelerate training of machine learning models, and are especially suited for deep learning.
GPUs make it possible to run models with large numbers of parameters quickly and efficiently. This is because GPUs can parallelize training tasks, distribute them across a large number of processors, and perform computational tasks concurrently.
Some data science teams acquire AI workstations, with multiple GPUs that provide huge concurrent processing power. Other teams take advantage of cloud-based compute instances with GPUs, which can be easily scaled up according to project needs without an upfront investment.
Learn more in the detailed guide to multi GPU
The first step in any ML or data science project is to find and collect the necessary data assets. However, the availability of adequate data remains one of the most common challenges facing organizations and data scientists, which directly impacts their ability to build robust ML models.
There are several reasons that data can be hard to collect and prepare for machine learning projects:
Data drift is one of the main reasons for the decrease of model accuracy over time. Data drift is a gradual change in input data which can affect model performance. Common reasons for data drift are:
Learn more in the detailed guide to model drift
Privacy concerns and growing compliance requirements make it difficult for data scientists to make use of datasets. Cybersecurity is becoming a bigger concern with the move to the public cloud. These two factors make it difficult for data scientists and machine learning teams to access the datasets they need.
Ensuring continued security and compliance with data protection regulations, such as the EU’s GDPR, presents an additional challenge for organizations. Datasets can contain personally identifiable information (PII), and failure to protect this data could result in severe financial penalties as well as pressure from regulators and costly audits.
Aporia is the machine learning observability platform, empowering data science and ML teams to trust their AI and act on Responsible AI principles. When a machine learning model starts interacting with the real world, making real predictions for real people and businesses, there are various triggers – like drift and model degradation – that can send your model spiraling out of control. Aporia is the best solution to ensure your ML models are optimized, working as intended, and showcasing value for the business.
Aporia fits naturally into your existing ML stack and seamlessly integrates with your existing ML infrastructure. Aporia delivers key features and tools for data science teams, ML teams, and business stakeholders to visualize, centralize, and improve their models in production:
Root Cause Investigation
To get a hands-on feel for Aporia’s ML monitoring solution, we recommend:
Book A Demo and get a guided tour to see Aporia in action
Authored by Aporia
Authored by Datagen
Authored by Datagen
Authored by Run.AI
Authored by Cynet
Authored by Staircase
Authored by Cloudinary
Authored by Brightsec