arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.07960cs.HCcs.CL
收藏

AgentClinic:一个用于评估AI在模拟临床环境中的多模态智能体基准

AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments

Stanford University(斯坦福大学);Johns Hopkins University(约翰霍普金斯大学);Stanford University Hospital Israelita Albert Einstein(斯坦福大学医院以色列阿尔伯特·爱因斯坦医院);ETH Zurich(苏黎世联邦理工学院)

Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, Michael Moor

AI总结:

AgentClinic提出多模态智能体基准,模拟临床环境评估LLM,发现序列决策显著降低诊断准确率,且工具使用能力差异大,Llama-3借助笔记工具提升达92%。

摘要:

在临床场景中评估大型语言模型(LLM)对于评估其潜在临床实用性至关重要。现有基准严重依赖静态问答,这无法准确描绘临床决策中复杂、连续的序列性特征。在此,我们引入AgentClinic,一个用于在模拟临床环境中评估LLM的多模态智能体基准,该环境包括患者交互、信息不完整条件下的多模态数据收集以及各种工具的使用,从而在九个医学专科和七种语言中进行深入评估。我们发现,以AgentClinic的序列决策格式解决MedQA问题难度显著增加,导致诊断准确率可降至原始准确率的十分之一以下。总体而言,我们观察到源自Claude-3.5的智能体在大多数设置中优于其他LLM骨干模型。然而,我们看到LLM在利用工具(如经验学习、自适应检索和反思循环)的能力上存在显著差异。引人注目的是,Llama-3在使用允许跨病例持久化编写和编辑笔记的笔记本工具时,相对改进高达92%。为进一步审视我们的临床模拟,我们利用真实世界电子健康记录,进行临床读者研究,用偏见干扰智能体,并探索这种交互环境首次启用的新颖患者中心指标。

英文摘要:

Evaluating large language models (LLM) in clinical scenarios is crucial to assessing their potential clinical utility. Existing benchmarks rely heavily on static question-answering, which does not accurately depict the complex, sequential nature of clinical decision-making. Here, we introduce AgentClinic, a multimodal agent benchmark for evaluating LLMs in simulated clinical environments that include patient interactions, multimodal data collection under incomplete information, and the usage of various tools, resulting in an in-depth evaluation across nine medical specialties and seven languages. We find that solving MedQA problems in the sequential decision-making format of AgentClinic is considerably more challenging, resulting in diagnostic accuracies that can drop to below a tenth of the original accuracy. Overall, we observe that agents sourced from Claude-3.5 outperform other LLM backbones in most settings. Nevertheless, we see stark differences in the LLMs' ability to make use of tools, such as experiential learning, adaptive retrieval, and reflection cycles. Strikingly, Llama-3 shows up to 92% relative improvements with the notebook tool that allows for writing and editing notes that persist across cases. To further scrutinize our clinical simulations, we leverage real-world electronic health records, perform a clinical reader study, perturb agents with biases, and explore novel patient-centric metrics that this interactive environment firstly enables.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑