arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.12422cs.CL
收藏

FactEHR: 一个用于使用LLMs评估临床笔记事实性的数据集

FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs

Khoury College of Computer Sciences, Northeastern University(东北大学克劳尔计算机科学学院);Center for Biomedical Informatics Research, Stanford University(斯坦福大学生物医学信息学研究中心);Department of Biomedical Data Science, Stanford School of Medicine(斯坦福医学院生物医学数据科学系);Stanford Health Care(斯坦福健康系统);Department of Medicine, Stanford School of Medicine(斯坦福医学院医学系);Clinical Excellence Research Center, Stanford School of Medicine(斯坦福医学院临床卓越研究中心);Department of Anesthesiology, Perioperative & Pain Medicine, Stanford School of Medicine(斯坦福医学院麻醉学、手术及疼痛医学系);Department of Dermatology, Stanford School of Medicine(斯坦福医学院皮肤病学系);Technology and Digital Solutions, Stanford Health Care(斯坦福健康系统技术与数字解决方案)

Monica Munnangi, Akshay Swaminathan, Jason Alan Fries, Jenelle Jindal, Sanjana Narayanan, Ivan Lopez, Lucia Tu, Philip Chung, Jesutofunmi A. Omiye, Mehr Kashyap, Nigam Shah

AI总结:

FactEHR是一个用于评估LLMs在临床笔记中事实分解能力的数据集,通过生成987,266个蕴含对揭示LLM在细粒度事实验证中的性能差异。

摘要:

验证和归因事实性声明对于安全有效地使用大型语言模型(LLMs)在医疗保健中至关重要。事实分解是事实性评估的核心组成部分,即将复杂的临床陈述分解为细粒度的原子事实以进行验证。最近的研究提出了事实分解,利用LLMs将源文本重写为简洁的句子,传达单一信息,以促进细粒度的事实验证。然而,临床文档在事实分解中提出了独特的挑战,由于术语密集和多样化的笔记类型,仍缺乏研究。为了解决这一差距并探索这些挑战,我们提出了FactEHR,一个包含2168份临床笔记的NLI数据集,涵盖三个医院系统中的四种类型,产生987,266个蕴含对。我们从LLMs的蕴含评估到定性分析的不同轴上评估生成的事实。我们的评估,包括临床医生的审查,揭示了LLM在事实分解中的表现存在显著差异。例如,Gemini-1.5-Flash始终生成相关且准确的事实,而Llama-3 8B生成较少且不一致的输出。结果强调了需要更好的LLM能力来支持临床文本中的事实验证。

英文摘要:

Verifying and attributing factual claims is essential for the safe and effective use of large language models (LLMs) in healthcare. A core component of factuality evaluation is fact decomposition, the process of breaking down complex clinical statements into fine-grained atomic facts for verification. Recent work has proposed fact decomposition, which uses LLMs to rewrite source text into concise sentences conveying a single piece of information, to facilitate fine-grained fact verification. However, clinical documentation poses unique challenges for fact decomposition due to dense terminology and diverse note types and remains understudied. To address this gap and explore these challenges, we present FactEHR, an NLI dataset consisting of document fact decompositions for 2,168 clinical notes spanning four types from three hospital systems, resulting in 987,266 entailment pairs. We assess the generated facts on different axes, from entailment evaluation of LLMs to a qualitative analysis. Our evaluation, including review by the clinicians, reveals substantial variability in LLM performance for fact decomposition. For example, Gemini-1.5-Flash consistently generates relevant and accurate facts, while Llama-3 8B produces fewer and less consistent outputs. The results underscore the need for better LLM capabilities to support factual verification in clinical text.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑