arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.10074cs.HC
收藏

STAGER 清单:用于评估生成式AI可靠性的标准化测试与评估指南

STAGER checklist: Standardized Testing and Assessment Guidelines for Evaluating Generative AI Reliability

Jinghong Chen, Lingxuan Zhu, Weiming Mou, Zaoqu Liu, Quan Cheng, Anqi Lin, Jian Zhang, Peng Luo

AI总结:

针对医疗领域生成式AI评估缺乏系统框架、现有研究方法设计有缺陷的问题,多学科团队通过文献综述和研讨制定了含23项条目的STAGER评估清单,提供标准化评估路径以提升研究质量,推动AI在医学生命科学领域发展。

摘要:

生成式人工智能(AI)在医疗应用中具备巨大潜力。已有大量研究探索了各类生成式AI模型在医疗场景中的效能,但目前仍缺乏全面、系统的评估框架。鉴于部分评估生成式AI医疗应用能力的研究在方法设计上存在缺陷,当前也缺少针对这类评估的标准化指南。为此,我们的目标是制定专门用于评估生成式AI系统在医疗场景中性能的标准化评估指南。为实现这一目标,我们利用PubMed和Google Scholar数据库开展了全面的文献综述,重点关注测试生成式AI在医学中能力的研究。我们的多学科团队由生命科学、临床医学、医学工程领域的专家以及生成式AI用户组成,经过多轮讨论后制定了包含23个条目的清单。该清单旨在全面覆盖医疗应用中生成式AI评估的关键维度。这份清单及其所支撑的更广泛评估框架涵盖了多个关键维度,包括问题收集、查询方法和评估技术,我们旨在实现对AI系统的整体性评估。该清单清晰勾勒出从问题收集到结果评估的完整路径,为研究人员应对潜在挑战与陷阱提供指导。我们的框架为测试生成式AI在医学中适用性的相关研究提供了一套标准化、系统化的方法,能够提升研究报告的质量,并助力生成式AI在医学与生命科学领域的发展。

英文摘要:

Generative Artificial Intelligence (AI) holds immense potential in medical applications. Numerous studies have explored the efficacy of various generative AI models within healthcare contexts, but there is a lack of a comprehensive and systematic evaluation framework. Given that some studies evaluating the ability of generative AI for medical applications have deficiencies in their methodological design, standardized guidelines for their evaluation are also currently lacking. In response, our objective is to devise standardized assessment guidelines tailored for evaluating the performance of generative AI systems in medical contexts. To this end, we conducted a thorough literature review using the PubMed and Google Scholar databases, focusing on research that tests generative AI capabilities in medicine. Our multidisciplinary team, comprising experts in life sciences, clinical medicine, medical engineering, and generative AI users, conducted several discussion sessions and developed a checklist of 23 items. The checklist is designed to encompass the critical evaluation aspects of generative AI in medical applications comprehensively. This checklist, and the broader assessment framework it anchors, address several key dimensions, including question collection, querying methodologies, and assessment techniques. We aim to provide a holistic evaluation of AI systems. The checklist delineates a clear pathway from question gathering to result assessment, offering researchers guidance through potential challenges and pitfalls. Our framework furnishes a standardized, systematic approach for research involving the testing of generative AI's applicability in medicine. It enhances the quality of research reporting and aids in the evolution of generative AI in medicine and life sciences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑