利用大语言模型从电子健康记录中检索证据:可能性与挑战
Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges
AI总结:
本文提出并评估了一种零样本LLM方法,用于从EHR非结构化记录中检索和总结与查询相关的证据,并通过LLM自动评估扩大实验规模,同时发现模型置信度可用于抑制幻觉。
摘要:
电子健康记录(EHR)中的非结构化数据往往包含关键信息——这些信息与影像互为补充——可以为放射科医生的诊断提供依据。但与患者相关的大量病历记录以及时间限制,使得人工识别相关证据在实践中几乎不可行。在这项工作中,我们提出并评估了一种零样本策略,将大语言模型(LLM)作为一种机制,用于高效检索和总结患者EHR中与给定查询相关的非结构化证据。我们的方法要求LLM根据相关病历记录推断患者是否患有某种疾病或存在患病风险;如果是,则要求模型总结支持性证据。在专家评估下,我们发现这种基于LLM的方法所提供的输出始终优于LLM出现之前的信息检索基线。人工评估成本高昂,因此我们还提出并验证了一种使用LLM来评估(其他)LLM在该任务上的输出的方法,从而使我们能够扩大评估规模。我们的研究结果表明,LLM作为EHR接口具有广阔前景,但也凸显了“幻觉”所带来的突出挑战。然而,在这一场景下,我们表明模型对输出的置信度与忠实摘要之间存在强相关性,这为限制虚构内容提供了一种实用手段。
英文摘要:
Unstructured data in Electronic Health Records (EHRs) often contains critical information -- complementary to imaging -- that could inform radiologists' diagnoses. But the large volume of notes often associated with patients together with time constraints renders manually identifying relevant evidence practically infeasible. In this work we propose and evaluate a zero-shot strategy for using LLMs as a mechanism to efficiently retrieve and summarize unstructured evidence in patient EHR relevant to a given query. Our method entails tasking an LLM to infer whether a patient has, or is at risk of, a particular condition on the basis of associated notes; if so, we ask the model to summarize the supporting evidence. Under expert evaluation, we find that this LLM-based approach provides outputs consistently preferred to a pre-LLM information retrieval baseline. Manual evaluation is expensive, so we also propose and validate a method using an LLM to evaluate (other) LLM outputs for this task, allowing us to scale up evaluation. Our findings indicate the promise of LLMs as interfaces to EHR, but also highlight the outstanding challenge posed by "hallucinations". In this setting, however, we show that model confidence in outputs strongly correlates with faithful summaries, offering a practical means to limit confabulations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。