arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.02559cs.CLcs.AI
收藏

基于文献综述的医疗领域大型语言模型人工评估框架

A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review

University of Pittsburgh(匹兹堡大学);University of Pittsburgh Medical Center(匹兹堡大学医学中心);University of Texas Health Houston(德克萨斯大学休斯顿健康科学中心);Cleveland Clinic(克利夫兰诊所);Keck School of Medicine, University of Southern California(南加州大学凯克医学院);Weill Cornell Medicine(威尔康奈尔医学院)
Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, Xizhi Wu, Shyam Visweswaran, S… 展开作者

Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, Xizhi Wu, Shyam Visweswaran, Sunyang Fu, Piyush Mathur, Giovanni E. Cacciamani, Cong Sun, Yifan Peng, Yanshan Wang

AI总结:

本研究回顾了医疗领域LLMs人工评估的文献,指出当前评估方法缺乏标准化,并基于PRISMA指南和现有策略提出了名为QUEST的全面实用人工评估框架,以提高评估的可靠性、普适性和适用性。

摘要:

随着生成式人工智能(AI),特别是大型语言模型(LLMs),持续在医疗领域取得进展,用人工评估补充传统自动化评估变得至关重要。理解和评估LLMs的输出对于确保安全性、可靠性和有效性至关重要。然而,人工评估繁琐、耗时且非标准化的特性,对在实践中全面评估和广泛采用LLMs构成了重大障碍。本研究回顾了关于医疗领域LLMs人工评估方法的现有文献。我们强调了对标准化且一致的人工评估方法的显著需求。我们遵循系统评价和荟萃分析首选报告项目(PRISMA)指南进行的广泛文献检索,涵盖了2018年1月至2024年2月的出版物。该综述考察了跨不同医学专科的LLMs人工评估,涉及评估维度、样本类型和规模、评估者的选择与招募、框架和指标、评估过程以及统计分析类型等因素。借鉴这些研究中采用的多样化评估策略,我们提出了一个全面且实用的LLMs人工评估框架:QUEST:信息质量、理解与推理、表达风格与角色、安全与伤害、以及信任与信心。该框架旨在通过定义明确的评估维度并提供详细的指南,提高LLMs人工评估在不同医疗应用中的可靠性、普适性和适用性。

英文摘要:

With generative artificial intelligence (AI), particularly large language models (LLMs), continuing to make inroads in healthcare, it is critical to supplement traditional automated evaluations with human evaluations. Understanding and evaluating the output of LLMs is essential to assuring safety, reliability, and effectiveness. However, human evaluation's cumbersome, time-consuming, and non-standardized nature presents significant obstacles to comprehensive evaluation and widespread adoption of LLMs in practice. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare. We highlight a notable need for a standardized and consistent human evaluation approach. Our extensive literature search, adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, includes publications from January 2018 to February 2024. The review examines the human evaluation of LLMs across various medical specialties, addressing factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Drawing on the diverse evaluation strategies employed in these studies, we propose a comprehensive and practical framework for human evaluation of LLMs: QUEST: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence. This framework aims to improve the reliability, generalizability, and applicability of human evaluation of LLMs in different healthcare applications by defining clear evaluation dimensions and offering detailed guidelines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑