基于非参数Bootstrap的可解释不确定性估计监测模型退化
Monitoring Model Deterioration with Explainable Uncertainty Estimation via Non-parametric Bootstrap
摘要：
部署机器学习模型后对其进行监控是一项挑战。当标记数据无法获取并且监控性能指标变得不可行时，决定何时在真实情况下重新训练模型更具挑战性。在这项工作中，我们使用非参数自举不确定性估计和SHAP值来提供可解释的不确定性估计，作为一种技术，旨在监控部署环境中机器学习模型的恶化，以及确定目标标签不可用时模型恶化的来源。经典方法纯粹是为了检测分布偏移，这可能导致假阳性，因为尽管数据分布发生了偏移，但模型没有恶化。为了估计模型的不确定性，我们使用一种新的自举方法来构建预测区间，该方法改进了Kumar & Srivastava（2012）的工作。我们表明，我们的模型恶化检测系统以及我们的不确定性估计方法实现了更好的性能比目前最先进的。最后，我们使用可解释的人工智能技术来了解模型恶化的驱动因素。我们发布了一个开源Python包doubt，它实现了我们提出的方法，以及用于重现我们实验的代码。
英文摘要：
Monitoring machine learning models once they are deployed is challenging. It is even more challenging to decide when to retrain models in real-case scenarios when labeled data is beyond reach, and monitoring performance metrics becomes unfeasible. In this work, we use non-parametric bootstrapped uncertainty estimates and SHAP values to provide explainable uncertainty estimation as a technique that aims to monitor the deterioration of machine learning models in deployment environments, as well as determine the source of model deterioration when target labels are not available. Classical methods are purely aimed at detecting distribution shift, which can lead to false positives in the sense that the model has not deteriorated despite a shift in the data distribution. To estimate model uncertainty we construct prediction intervals using a novel bootstrap method, which improves upon the work of Kumar & Srivastava (2012). We show that both our model deterioration detection system as well as our uncertainty estimation method achieve better performance than the current state-of-the-art. Finally, we use explainable AI techniques to gain an understanding of the drivers of model deterioration. We release an open source Python package, doubt, which implements our proposed methods, as well as the code used to reproduce our experiments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
