1. Intro

• ML Observability: Evaluate machine learning model performance in production
• MLOps is Not Always DevOps

2. ML evaluation metrics

• ROC-AUC
• Precision
• Recall
• F1 Score
• Binary Cross-Entropy
• RMSE
• MAPE
• PR AUC
• NDCG

3. Drift Metrics

• PSI
• Concept Drift: 8 Detection Methods

4. Fairness metrics

• Common Fairness Metrics

5. Explainability

• SHAP
• Permutation Importance (PI)
• Feature Importance: 7 Methods

6. Production ML

• Outliers in A/B Testing
• Data sampling
• Optimizing ad placement in search & recommendation systems
• Entity-level monitoring

7. LLM Observability

• Embeddings
• Monitoring LLMs: Metrics, challenges, & hallucinations