电子健康记录信息检索的活基准

中文摘要 AI 辅助

基于大型语言模型（LLM）的临床助手正日益集成到电子健康记录（EHR）系统中，改变了临床医生从患者记录中检索和综合信息的方式。其安全性和实用性依赖于严格的评估，然而现有基准是人工策划的，更新成本高昂，并且随着技术的快速发展迅速过时。我们提出了一个可扩展的框架，能够从纵向EHR笔记中自动生成问答对。十九位临床医生验证了基准生成器，产生了电子健康记录信息检索基准（BRIE），这是一个可持续维护的评估数据集。在九个LLM和五种推理策略中，最先进的系统经常遗漏临床重要信息，尤其是在需要跨多个文档和就诊进行综合的问题上。由于生成器本身经过验证，BRIE支持静态基准无法提供的评估，包括生成反映临床医生推理差异的多个答案以进行稳健的性能评估，以及持续刷新基准内容以防止数据泄漏。我们的结果表明，可扩展的基准生成能够对部署在快速发展的医疗环境中的临床LLM进行严格且最新的评估。

英文摘要

Large language model (LLM)-based clinical assistants are increasingly being integrated into electronic health record (EHR) systems, transforming how clinicians retrieve and synthesize information from patient records. Their safety and utility depend on rigorous evaluation, yet existing benchmarks are manually curated, costly to update, and rapidly become obsolete with evolving technological advancements. We present a scalable framework that automatically generates question--answer pairs from longitudinal EHR notes. Nineteen clinicians validate the benchmark generator, producing the Benchmark for Retrieving Information in EHRs (BRIE), a continuously maintainable evaluation dataset. Across nine LLMs and five inference strategies, state-of-the-art systems frequently omit clinically important information, particularly for questions requiring synthesis across multiple documents and encounters. Because the generator itself is validated, BRIE supports evaluations that static benchmarks cannot, including the generation of multiple answers that reflect variation in clinician reasoning for robust performance assessment and continuously refreshing benchmark content to guard against leakage. Our results demonstrate that scalable benchmark generation enables rigorous, up-to-date evaluation of clinical LLMs as they are deployed in rapidly evolving healthcare settings.

发表机构

• Stanford University（斯坦福大学）
• Stanford Cancer Center（斯坦福癌症中心）
• Weill Cancer Hub West（威尔癌症中心西部分中心）
• Stanford School of Medicine（斯坦福医学院）

机构由 AI 辅助整理，请以论文原文为准。