arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.12347cs.LGcs.AIcs.CVeess.IV
收藏

故障分析与模型修复的原理性方法:在医学影像中的应用

A Principled Approach to Failure Analysis and Model Repairment: Demonstration in Medical Imaging

Rokken Inc.(Rokken公司);Terumo Corporation, Corporate R&D Center(泰尔茂公司企业研发中心);Department of Computing, Imperial College London(帝国理工学院计算机系);Department of Computing, University College London(伦敦大学学院计算机系)

Thomas Henn, Yasukazu Sakamoto, Clément Jacquet, Shunsuke Yoshizawa, Masamichi Andou, Stephen Tchen, Ryosuke Saga, Hiroyuki Ishihara, Katsuhiko Shimizu, Yingzhen Li, Ryutaro Tanno

摘要:

机器学习模型通常会在部署后出现意外故障,这是由于数据偏移或训练环境中的不常见情况造成的。领域专家通常会经历手动检查故障案例、识别故障模式、然后尝试修复模型的繁琐过程。在这项工作中,我们的目标是通过回答两个关键问题来解释并为这个过程带来原则:(i)我们如何知道我们已经识别了有意义的和不同的故障类型?(ii)我们如何验证一个模型是否确实得到了修复?我们建议,可以通过测量内部和类型间的泛化后微调和引入指标来比较不同的分型方法的质量进行验证,确定的故障类型。此外,我们认为,一个模型可以被认为是修复的,如果它实现了高精度的故障类型,同时保留性能的先前正确的数据。我们将这两个想法结合成一个原则性的框架,用于评估已识别的故障子类型和模型修复的质量。我们评估其效用的分类和目标检测任务。我们的代码可在https://github.com/Rokken-lab6/Failure-Analysis-and-Model-Repairment上获得

英文摘要:

Machine learning models commonly exhibit unexpected failures post-deployment due to either data shifts or uncommon situations in the training environment. Domain experts typically go through the tedious process of inspecting the failure cases manually, identifying failure modes and then attempting to fix the model. In this work, we aim to standardise and bring principles to this process through answering two critical questions: (i) how do we know that we have identified meaningful and distinct failure types?; (ii) how can we validate that a model has, indeed, been repaired? We suggest that the quality of the identified failure types can be validated through measuring the intra- and inter-type generalisation after fine-tuning and introduce metrics to compare different subtyping methods. Furthermore, we argue that a model can be considered repaired if it achieves high accuracy on the failure types while retaining performance on the previously correct data. We combine these two ideas into a principled framework for evaluating the quality of both the identified failure subtypes and model repairment. We evaluate its utility on a classification and an object detection tasks. Our code is available at https://github.com/Rokken-lab6/Failure-Analysis-and-Model-Repairment

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑