Working With Amazon SageMaker Model Monitor

What Is Amazon SageMaker? 

Amazon SageMaker is a fully managed machine learning service offered by AWS (Amazon Web Services) that allows data scientists and machine learning engineers to easily create, train, and deploy ML models. It offers a seamless experience by integrating Jupyter notebooks, built-in algorithms, model training infrastructure, and real-time prediction capabilities.

Key features of Amazon SageMaker include:

• Jupyter notebooks: Interactive computing environments that simplify data exploration, preprocessing, and model development using ML frameworks like TensorFlow or PyTorch.
• Built-in algorithms: Covering common use cases such as classification, regression analysis, and time series forecasting, saving time on creating custom solutions.
• Distributed model training: Powerful distributed training capabilities with GPU instances for faster computation, allowing you to scale up model performance without handling infrastructure management tasks.
• A/B testing and deployment: Easy deployment of trained models with A/B testing support to compare different versions of an algorithm, and automatic scaling based on incoming traffic, ensuring optimal resource utilization.

Amazon SageMaker provides SageMaker Model Monitor, a service that lets you monitor the effectiveness and performance of machine learning models in production.

This is part of a series of articles about Machine Learning Models. 

What Is AWS SageMaker Model Monitor? 

Amazon SageMaker Model Monitor helps detect and diagnose model quality issues, data drift, and concept drift, which are common challenges in machine learning model deployment. It enables continuous monitoring of production models and sends alerts when issues arise, allowing for prompt corrective action.

SageMaker Model Monitor works by analyzing the input and output data of deployed models and comparing them to the training data. It employs statistical analysis and machine learning techniques to identify anomalies and deviations from expected behavior. You can configure it to track specific metrics and set up alerts based on defined thresholds.

Key features of SageMaker Model Monitor include:

• Data quality monitoring: Ensures input data is consistent with the training dataset by detecting changes in statistical properties or missing features.
• Model quality monitoring: Tracks model performance on new incoming data by monitoring key performance metrics like accuracy, precision, recall, and F1 score, indicating when retraining or tuning is needed.
• Bias drift detection: Maintains fairness in AI systems by identifying potential biases introduced during training or due to changes in real-world conditions after deployment.
• Feature attribution drift monitoring: Monitors shifts in feature importance over time, which may impact the prediction accuracy and stability of ML models.

Learn more in our detailed guide to model monitoring 

Benefits of Using AWS SageMaker Model Monitor 

Using Amazon SageMaker Model Monitor in your machine learning pipeline provides several advantages for both development teams and businesses:

• Maintaining high-quality models: Ensures deployed models continue to provide accurate predictions even as underlying patterns change or new inputs arrive.
• Automated monitoring and alerting: Enabling setting up automated monitoring schedules to check for data drift, model performance degradation, or bias issues, allowing proactive problem-solving.
• Faster troubleshooting: Detailed reports on detected issues enable engineers to identify root causes and take corrective actions quickly.

Working With Amazon SageMaker Model Monitor 

Monitor Data Quality

Data quality is crucial for maintaining the accuracy and effectiveness of machine learning models. Amazon SageMaker Model Monitor provides a comprehensive solution for automatically monitoring input data integrity. 

Here is a general process for working with SageMaker Model Monitor to analyze data quality:

1. Create a baseline dataset using historical data from training sets or other sources.
2. Analyze the baseline dataset by calculating descriptive statistics and generating schema information.
3. Create a monitoring schedule to regularly capture new datasets from live endpoints.
4. Analyze captured datasets against established baselines using AWS-provided or custom algorithms.
5. Receive alerts via CloudWatch Events or SNS notifications if discrepancies are detected.

Monitor Model Quality

It is important to continuously assess model performance after deployment. Amazon SageMaker Model Monitor tracks various metrics and compares them against predefined thresholds:

• Model quality monitoring: Lets you track deployed model performance and receive alerts if quality degrades.
• Bias metric frameworks: Use pre-built or custom bias metrics to assess fairness across different demographic groups.

Monitor Bias Drift for Models in Production

Bias drift occurs when model predictions become biased towards specific demographics or subgroups. Model Monitor helps detect and mitigate bias drift by continuously monitoring key fairness metrics:

1. Create a baseline dataset with ground truth labels, including demographic information.
2. Analyze the baseline dataset using AWS-provided or custom fairness algorithms.
3. Create a monitoring schedule for capturing new datasets from live endpoints with ground truth labels.
4. Analyze captured datasets against baselines to identify significant deviations in fairness metrics, indicating potential bias drift issues.

Monitor Feature Attribution Drift for Models in Production

A shift in the distribution of real-time data for models in production can lead to a corresponding shift in feature attribution values, similar to how it might cause a drift in bias when monitoring bias metrics. You can use Amazon SageMaker’s Clarify feature track predictions for feature attribution drift. Here is a general process for working with feature attribution drift:

1. View feature attribution: While the model is being monitored, view exportable reports and charts displaying feature attributions in SageMaker Studio.
2. Set up alerts in Amazon CloudWatch: Configure alerts in Amazon CloudWatch to receive notifications if the attribution values exceed a predetermined threshold.
3. Detect drift by comparing feature rankings: Compare the changes in the ranking of individual features from the training data to the real-time data to detect drift. 
4. Analyze feature rankings: be sensitive to both alterations in ranking order and the raw attribution score of the features when detecting drift. If two features experience the same drop in ranking from training to real-time data, prioritize the feature with a higher attribution score during the training phase.
5. Use Normalized Discounted Cumulative Gain (NDCG) score: Employ the NDCG score to compare the feature attribution rankings of training and real-time data.

Aporia: A SageMaker Model Monitor Alternative

Aporia’s cutting-edge ML observability platform offers seamless integration with AWS SageMaker models, providing a powerful alternative to SageMaker Model Monitor. This innovative solution simplifies the process of monitoring and maintaining machine learning models by automatically generating advanced custom monitoring templates tailored to your specific use case. 

In addition, Aporia’s platform allows for swift anomaly detection, drift monitoring, and performance analysis, ensuring your SageMaker models continue to deliver accurate predictions and maintain optimal performance. By leveraging Aporia’s user-friendly interface and advanced analytics, data scientists and ML engineers can easily gain insights into their models’ behavior, streamline debugging, and efficiently address any performance issues, all without the need for extensive manual intervention or code modification.

Aporia empowers organizations with key features and tools to ensure high model performance and Responsible AI: 

Model Visibility

• Single pane of glass visibility into all production models. Custom dashboards that can be understood and accessed by all relevant stakeholders.
• Track model performance and health in one place. 
• A centralized hub for all your models in production.
• Custom metrics and widgets to ensure you’re getting the insights that matter to you.

ML Monitoring

• Start monitoring in minutes.
• Instant alerts and advanced workflows trigger. 
• Custom monitors to detect data drift, model degradation, performance, etc.
• Track relevant custom metrics to ensure your model is drift-free and performance is driving value. 
• Choose from our automated monitors or get hands-on with our code-based monitor options. 

Explainable AI

• Get human-readable insight into your model predictions. 
• Simulate ‘What if?’ situations. Play with different features and find how they impact predictions.
• Gain valuable insights to optimize model performance.
• Communicate predictions to relevant stakeholders and customers.

Root Cause Investigation

• Slice and dice model performance, data segments, data stats, or distribution.
• Identify and debug issues.
• Explore and understand connections in your data.

To get a hands-on feel for Aporia’s advanced model monitoring and deep model visualization tools, we recommend:

• Book a demo to get a guided tour of Aporia.
• Start your free trial with Aporia.

Gon Rappaport