A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review
University of Pittsburgh(匹兹堡大学);University of Pittsburgh Medical Center(匹兹堡大学医学中心);University of Texas Health Houston(德克萨斯大学休斯顿健康科学中心);Cleveland Clinic(克利夫兰诊所);Keck School of Medicine, University of Southern California(南加州大学凯克医学院);Weill Cornell Medicine(威尔康奈尔医学院)
Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, Xizhi Wu, Shyam Visweswaran, S… 展开作者
Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, Xizhi Wu, Shyam Visweswaran, Sunyang Fu, Piyush Mathur, Giovanni E. Cacciamani, Cong Sun, Yifan Peng, Yanshan Wang
With generative artificial intelligence (AI), particularly large language models (LLMs), continuing to make inroads in healthcare, it is critical to supplement traditional automated evaluations with human evaluations. Understanding and evaluating the output of LLMs is essential to assuring safety, reliability, and effectiveness. However, human evaluation's cumbersome, time-consuming, and non-standardized nature presents significant obstacles to comprehensive evaluation and widespread adoption of LLMs in practice. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare. We highlight a notable need for a standardized and consistent human evaluation approach. Our extensive literature search, adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, includes publications from January 2018 to February 2024. The review examines the human evaluation of LLMs across various medical specialties, addressing factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Drawing on the diverse evaluation strategies employed in these studies, we propose a comprehensive and practical framework for human evaluation of LLMs: QUEST: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence. This framework aims to improve the reliability, generalizability, and applicability of human evaluation of LLMs in different healthcare applications by defining clear evaluation dimensions and offering detailed guidelines.
A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promises
Emory University(埃默里大学);University of Michigan(密歇根大学);Georgia Institute of Technology(佐治亚理工学院);Rensselaer Polytechnic Institute(伦斯勒理工学院);Case Western Reserve University(凯斯西储大学);Iowa State University(爱荷华州立大学);Weill Cornell Medicine, Cornell University(康奈尔大学威尔医学院)
Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review
Peking University(北京大学);Peking University Cancer Hospital and Institute(北京大学肿瘤医院暨北京市肿瘤防治研究所);National Engineering Laboratory for Big Data Analysis and Applications, Peking University(北京大学大数据分析与应用技术国家工程实验室);Peking University Changsha Institute for Computing and Digital Economy(北京大学长沙计算与数字经济研究院)
Privacy Challenges and Solutions in Retrieval-Augmented Generation-Enhanced LLMs for Healthcare Chatbots: A Review of Applications, Risks, and Future Directions
Centre for Smart Health, School of Nursing(智能健康中心、护理学院);The Hong Kong Polytechnic University(香港理工大学);Department of Electrical and Electronic Engineering(电子与电气工程系);Faculty of Health Sciences(健康科学学院);University of Maribor(马里博大学)
A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination
Biomedical Informatics and Medical Education(生物医学信息学与医学教育);Health AI(健康人工智能);Department of Information Sciences and Technology(信息科学与技术系);Department of Linguistics(语言学系)
A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework
Department of Biomedical Informatics, School of Medicine, University of Pittsburgh(匹兹堡大学医学院生物医学信息学系);Department of Health Information Management, School of Health and Rehabilitation Sciences, University of Pittsburgh(匹兹堡大学健康与康复科学学院健康信息管理系);OpenCura, Health Innovation Consortium(OpenCura健康创新联盟);Northwestern University, Kellogg School of Management(西北大学凯洛格管理学院);Intelligent Systems Program, School of Computing and Information, University of Pittsburgh(匹兹堡大学计算与信息学院智能系统项目);Johns Hopkins University School of Medicine Biomedical Informatics and Data Science(约翰霍普金斯大学医学院生物医学信息学与数据科学);Clinical and Translational Science Institute, University of Pittsburgh(匹兹堡大学临床与转化科学研究所);Institute for Biomedical Informatics, University of Pennsylvania(宾夕法尼亚大学生物医学信息学研究所);Data Science, School of Computing and Information, University of Pittsburgh(匹兹堡大学计算与信息学院数据科学)
Prompt Engineering for Healthcare: Methodologies and Applications
School of Computer Science, Northwestern Polytechnical University(西北工业大学计算机学院);School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);School of Computing, The University of Georgia(佐治亚大学计算学院);Department of Computer Science and Engineering, Lehigh University(利哈伊大学计算机科学与工程系);Department of Radiology, Massachusetts General Hospital and Harvard Medical School(麻省总医院与哈佛医学院放射科);Department of Computer Science and Engineering, The University of Texas at Arlington(德克萨斯大学阿灵顿分校计算机科学与工程系);School of Biomedical Engineering, ShanghaiTech University(上海科技大学生物医学工程学院);Shanghai United Imaging Intelligence Co., Ltd.(上海联影智能医疗科技有限公司);Shanghai Clinical Research and Trial Center(上海临床研究与试验中心);Department of Electronic Engineering, City University of Hong Kong(香港城市大学电子工程系)
Artificial Intelligence in Rural Healthcare Delivery: Bridging Gaps and Enhancing Equity through Innovation
University of Illinois College of Medicine Rockford(伊利诺伊大学罗克福德医学院);National Center for Rural Health Professions(农村健康专业中心);Sri Sivasubramaniya Nadar College of Engineering(塞维萨布拉曼尼亚那德尔工程学院);M. Kumarasamy College of Engineering(M. Kumarasamy 工程学院);The Ohio State University Wexner Medical Center(俄亥俄州立大学韦克斯纳医疗中心);iWorks Corporation(iWorks 公司);Central South University(中南大学);Bacha Khan University(巴哈汗大学)
North Africa's Missing Framework: NLP-Driven Mental Healthcare in Algeria and Implications for Low-resource Settings
University Mohamed El Bachir El Ibrahimi of Bordj Bou Arreridj(布吉布阿雷里吉·穆罕默德·巴希尔·易卜拉欣大学);Laboratory of Informatics and its Applications of M’sila(姆西拉信息学及其应用实验室);Laboratory of Materials and Electronic Systems(材料与电子系统实验室)
Natural Language Processing in Electronic Health Records in Relation to Healthcare Decision-making: A Systematic Review
University of Southern Queensland(南昆士兰大学);Queensland University of Technology(昆士兰科技大学);Metro North Mental Health(北大都会心理健康中心);University of Rochester(罗切斯特大学);North South University(南北大学)