弥合可重复性鸿沟:开源软件在标准化医疗AI中的作用
Bridging the Reproducibility Divide: Open Source Software's Role in Standardizing Healthcare AI
AI总结:
本文探讨了开源软件在提升医疗AI可重复性中的作用,指出通过公开数据集和代码共享可显著提高论文影响力,并提出标准化数据预处理和开放科学实践的必要性。
摘要:
我们对最近的AI4H出版物进行分析发现,尽管存在使用公开数据集和分享建模代码的趋势,但74%的AI4H论文仍依赖于私人数据集或不分享其代码。这在医疗应用中尤其令人担忧,因为在医疗应用中信任至关重要。此外,不一致且文档不全的数据预处理管道导致相同任务和数据集上模型性能报告的差异,使得难以评估AI模型的真实有效性。尽管可重复性危机带来的挑战,通过开放实践来解决这些问题提供了显著的好处。例如,尽管可重复性要求增加了研究和出版的额外努力,但它显著提高了工作的影响力。我们的分析显示,使用公开数据集并分享代码的论文平均获得比既不使用也不分享代码的论文多110%的引用,这使引用数量翻倍以上。鉴于增强可重复性的明显好处,AI4H社区必须采取切实步骤克服现有障碍。该社区应推广开放科学实践,建立标准化的数据预处理指南,并开发稳健的基准。通过开源开发解决这些挑战可以提高可重复性,这对于确保AI模型安全、有效并有益于患者护理至关重要。这种方法将帮助构建更可信的AI系统,能够整合到医疗环境中,最终改善患者结果并推动医学领域的发展。
英文摘要:
Our analysis of recent AI4H publications reveals that, despite a trend toward utilizing open datasets and sharing modeling code, 74% of AI4H papers still rely on private datasets or do not share their code. This is especially concerning in healthcare applications, where trust is essential. Furthermore, inconsistent and poorly documented data preprocessing pipelines result in variable model performance reports, even for identical tasks and datasets, making it challenging to evaluate the true effectiveness of AI models. Despite the challenges posed by the reproducibility crisis, addressing these issues through open practices offers substantial benefits. For instance, while the reproducibility mandate adds extra effort to research and publication, it significantly enhances the impact of the work. Our analysis shows that papers that used both public datasets and shared code received, on average, 110% more citations than those that do neither--more than doubling the citation count. Given the clear benefits of enhancing reproducibility, it is imperative for the AI4H community to take concrete steps to overcome existing barriers. The community should promote open science practices, establish standardized guidelines for data preprocessing, and develop robust benchmarks. Tackling these challenges through open-source development can improve reproducibility, which is essential for ensuring that AI models are safe, effective, and beneficial for patient care. This approach will help build more trustworthy AI systems that can be integrated into healthcare settings, ultimately contributing to better patient outcomes and advancing the field of medicine.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。