FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs
Khoury College of Computer Sciences, Northeastern University(东北大学克劳尔计算机科学学院);Center for Biomedical Informatics Research, Stanford University(斯坦福大学生物医学信息学研究中心);Department of Biomedical Data Science, Stanford School of Medicine(斯坦福医学院生物医学数据科学系);Stanford Health Care(斯坦福健康系统);Department of Medicine, Stanford School of Medicine(斯坦福医学院医学系);Clinical Excellence Research Center, Stanford School of Medicine(斯坦福医学院临床卓越研究中心);Department of Anesthesiology, Perioperative & Pain Medicine, Stanford School of Medicine(斯坦福医学院麻醉学、手术及疼痛医学系);Department of Dermatology, Stanford School of Medicine(斯坦福医学院皮肤病学系);Technology and Digital Solutions, Stanford Health Care(斯坦福健康系统技术与数字解决方案)
Monica Munnangi, Akshay Swaminathan, Jason Alan Fries, Jenelle Jindal, Sanjana Narayanan, Ivan Lopez, Lucia Tu, Philip Chung, Jesutofunmi A. Omiye, Mehr Kashyap, Nigam Shah
备注:
To appear at MLHC 2025
发表信息:
Journal-ref: Proceedings of Machine Learning Research (PMLR), vol. 298, Proceedings of the Machine Learning for Healthcare Conference (MLHC), 2025
Verifying and attributing factual claims is essential for the safe and effective use of large language models (LLMs) in healthcare. A core component of factuality evaluation is fact decomposition, the process of breaking down complex clinical statements into fine-grained atomic facts for verification. Recent work has proposed fact decomposition, which uses LLMs to rewrite source text into concise sentences conveying a single piece of information, to facilitate fine-grained fact verification. However, clinical documentation poses unique challenges for fact decomposition due to dense terminology and diverse note types and remains understudied. To address this gap and explore these challenges, we present FactEHR, an NLI dataset consisting of document fact decompositions for 2,168 clinical notes spanning four types from three hospital systems, resulting in 987,266 entailment pairs. We assess the generated facts on different axes, from entailment evaluation of LLMs to a qualitative analysis. Our evaluation, including review by the clinicians, reveals substantial variability in LLM performance for fact decomposition. For example, Gemini-1.5-Flash consistently generates relevant and accurate facts, while Llama-3 8B produces fewer and less consistent outputs. The results underscore the need for better LLM capabilities to support factual verification in clinical text.
ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents
Multiverse Computing;Parque Cientifico y Tecnológico de Gipuzkoa(吉普斯夸科技园);Centre for Social Innovation(社会创新中心);Donostia International Physics Center(多诺斯蒂亚国际物理中心);Ikerbasque Foundation for Science(伊克尔巴斯克科学基金会)
Structured Outputs Enable General-Purpose LLMs to be Medical Experts
University of Bristol(布里斯托大学);Lehigh University(利哈伊大学);The University of Queensland(昆士兰大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of California, Merced(加利福尼亚大学默塞德分校)
LEME: Open Large Language Models for Ophthalmology with Advanced Reasoning and Clinical Validation
Yale University(耶鲁大学);National University of Singapore(新加坡国立大学);Harvard Medical School(哈佛医学院);Korea University(高丽大学);Singapore Eye Research Institute(新加坡眼科研究所);Singapore National Eye Centre(新加坡国家眼科中心);National University Hospital(国立大学医院);University of Liverpool(利物浦大学);Stanford University(斯坦福大学);University College London(伦敦大学学院);Moorfields Eye Hospital(莫菲尔德眼科医院);National Cancer Institute(美国国家癌症研究所);National Institutes of Health(美国国立卫生研究院)
Systematic Evaluation of the Quality of Synthetic Clinical Notes Rephrased by LLMs at Million-Note Scale
Australian e-Health Research Centre, CSIRO, Australia(澳大利亚电子健康研究中心,CSIRO,澳大利亚);National Library of Medicine, National Institutes of Health, USA(国家医学图书馆,国立卫生研究院,美国)
Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance
Aalto University(阿尔托大学);Tencent(腾讯);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Hong Kong Polytechnic University(香港理工大学);Aarhus University(奥胡斯大学);Technical University of Munich(慕尼黑工业大学)
Integrating UMLS Knowledge into Large Language Models for Medical Question Answering
National University of Singapore(新加坡国立大学);Duke-NUS Medical School(杜克-新加坡国立大学医学院);University of Tokyo(东京大学);National Institute of Advanced Industrial Science and Technology(日本产业技术综合研究所);Singapore General Hospital(新加坡中央医院);Zhejiang Lab(之江实验室);National Institutes of Health(美国国立卫生研究院);Yale University(耶鲁大学)
Graph Alignment Topology as an Inductive Bias for Grounding Detection
Department of Pediatrics, University of Illinois College of Medicine Peoria(伊利诺伊大学皮奥里亚医学院儿科部);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校计算机与数据科学学院);Carle Illinois College of Medicine, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校卡莱医学院)
KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques
Texas A&M University(德克萨斯农工大学);The University of Tokyo(东京大学);Northwestern University(西北大学);Singapore General Hospital(新加坡中央医院);Zhejiang University(浙江大学);Zhejiang Lab(之江实验室);Yale University(耶鲁大学);National Institutes of Health(美国国立卫生研究院);Duke-NUS Medical School(杜克-新加坡国立大学医学院)
PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology
Department of Biostatistics, University of Washington(华盛顿大学生物统计学系);Clinical Research Division, Fred Hutch Cancer Center(Fred Hutch癌症中心临床研究部);Division of Hematology and Oncology, Department of Medicine, University of Washington(华盛顿大学医学系血液学与肿瘤学分会);Allen Institute for AI(Allen人工智能研究所);Department of Computer Science and Engineering, University of Washington(华盛顿大学计算机科学与工程系);Public Health Sciences, Biostatistics, Fred Hutchinson Cancer Center(Fred Hutchinson癌症中心公共卫生科学与生物统计学)