arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.16900cs.LGcs.AIcs.CY
收藏

规范AI适应:AI医疗设备更新分析

Regulating AI Adaptation: An Analysis of AI Medical Device Updates

Stanford University(斯坦福大学)

Kevin Wu, Eric Wu, Kit Rodolfa, Daniel E. Ho, James Zou

AI总结:

本文系统分析了FDA批准的AI医疗设备更新现状,发现不到2%设备通过重新训练更新,近四分之一通过新功能更新;案例研究表明特定数据重训可恢复性能但损害原站点表现,挑战了“一刀切”监管模式。

摘要:

尽管近年来AI的发展速度迅速推进,但安全有效的监管框架的制定却相对滞后。特别是,AI模型的适应性特征给监管机构带来了独特的挑战,因为更新模型可以提高其性能,但也可能引入安全风险。在美国,食品药品监督管理局(FDA)一直是监管和批准数百种AI医疗设备的先行者。为了更好地理解AI如何更新及其监管考量,我们系统分析了FDA批准的AI医疗设备更新的频率和性质。我们发现,所有设备中不到2%报告称已通过在新数据上重新训练进行更新。同时,近四分之一的设备报告了以新功能和市场声明形式的更新。作为说明性案例研究,我们分析了气胸检测模型,发现当在新站点评估时,模型性能可能下降高达0.18 AUC,而使用特定站点数据进行重新训练可以缓解这种性能下降,恢复高达0.23 AUC。然而,我们也观察到,在使用来自新站点的数据进行重新训练后,原始站点的性能显著下降,这从一个例子中提供了对当前“一刀切”监管审批方法的挑战的见解。我们的分析深入探讨了FDA批准的AI设备更新的现状,并为未来关于模型更新和适应性AI的监管政策提供了见解。

英文摘要:

While the pace of development of AI has rapidly progressed in recent years, the implementation of safe and effective regulatory frameworks has lagged behind. In particular, the adaptive nature of AI models presents unique challenges to regulators as updating a model can improve its performance but also introduce safety risks. In the US, the Food and Drug Administration (FDA) has been a forerunner in regulating and approving hundreds of AI medical devices. To better understand how AI is updated and its regulatory considerations, we systematically analyze the frequency and nature of updates in FDA-approved AI medical devices. We find that less than 2% of all devices report having been updated by being re-trained on new data. Meanwhile, nearly a quarter of devices report updates in the form of new functionality and marketing claims. As an illustrative case study, we analyze pneumothorax detection models and find that while model performance can degrade by as much as 0.18 AUC when evaluated on new sites, re-training on site-specific data can mitigate this performance drop, recovering up to 0.23 AUC. However, we also observed significant degradation on the original site after re-training using data from new sites, providing insight from one example that challenges the current one-model-fits-all approach to regulatory approvals. Our analysis provides an in-depth look at the current state of FDA-approved AI device updates and insights for future regulatory policies toward model updating and adaptive AI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑