AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments
Stanford University(斯坦福大学);Johns Hopkins University(约翰霍普金斯大学);Stanford University Hospital Israelita Albert Einstein(斯坦福大学医院以色列阿尔伯特·爱因斯坦医院);ETH Zurich(苏黎世联邦理工学院)
Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, Michael Moor
Evaluating large language models (LLM) in clinical scenarios is crucial to assessing their potential clinical utility. Existing benchmarks rely heavily on static question-answering, which does not accurately depict the complex, sequential nature of clinical decision-making. Here, we introduce AgentClinic, a multimodal agent benchmark for evaluating LLMs in simulated clinical environments that include patient interactions, multimodal data collection under incomplete information, and the usage of various tools, resulting in an in-depth evaluation across nine medical specialties and seven languages. We find that solving MedQA problems in the sequential decision-making format of AgentClinic is considerably more challenging, resulting in diagnostic accuracies that can drop to below a tenth of the original accuracy. Overall, we observe that agents sourced from Claude-3.5 outperform other LLM backbones in most settings. Nevertheless, we see stark differences in the LLMs' ability to make use of tools, such as experiential learning, adaptive retrieval, and reflection cycles. Strikingly, Llama-3 shows up to 92% relative improvements with the notebook tool that allows for writing and editing notes that persist across cases. To further scrutinize our clinical simulations, we leverage real-world electronic health records, perform a clinical reader study, perturb agents with biases, and explore novel patient-centric metrics that this interactive environment firstly enables.
ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院);Houston Methodist Hospital(休斯顿卫理公会医院);Trinity Health Group(三一健康集团);Stanford Oncology Partners(斯坦福肿瘤学伙伴);St. Luke Hospital(圣卢克医院)
本研究提出 ResidencyRL,通过多轮强化学习训练临床 AI 智能体,在模拟临床环境中提升诊断准确性、降低漏报率,且能力可迁移至多个医学基准测试,为临床 AI 发展提供了新路径。
MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models
ETH Zurich(苏黎世联邦理工学院);MBZUAI(马克斯·普朗克人工智能研究所);Amazon(亚马逊);University of Oxford(牛津大学);University of Bern(伯尔尼大学);Imperial College London(伦敦帝国理工学院);Peking University(北京大学)
Large Language Models as Unified Multimodal Learners for Clinical Prediction
German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));Charité Universitätsmedizin Berlin(柏林夏里特大学医学中心);DNC Information Management GmbH(DNC信息管理有限公司);Klinik für Akut- und Notfallmedizin, Asklepios Klinikum Harburg(哈堡阿斯克勒庇俄斯医院急性与急诊医学科);Technical University Berlin(柏林工业大学)
MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large Language Models
National Yang Ming Chiao Tung University(国立阳明交通大学);University of Michigan(密歇根大学);Stevens Institute of Technology(史蒂文斯理工学院);Massachusetts Institute of Technology(麻省理工学院);Far Eastern Memorial Hospital(远东纪念医院)
RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Macau University of Science and Technology(澳门科技大学);First Affiliated Hospital of Guangzhou Medical University(广州医科大学附属第一医院);Guangzhou Institute of Respiratory Health(广州呼吸健康研究院)
Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Models
Hong Kong Baptist University(香港 Baptist 大学);Shanxi University(山西大学);Shanghai Institute for Advanced Study of Zhejiang University(浙江大学上海先进研究院);Hong Kong University of Science and Technology(香港科技大学);The University of Manchester(曼彻斯特大学)
Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
Center for Biomedical Data Science, Duke-NUS Medical School(生物医学数据科学中心,杜克 - 新加坡国立大学医学院);Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School(杜克 - 新加坡国立大学人工智能与医学科学计划,杜克 - 新加坡国立大学医学院);Department of Population Health Sciences, Weill Cornell Medicine(人口健康科学系,威尔康奈尔医学院);System Engineering, College of Engineering, Cornell University(系统工程,康奈尔大学工程学院);Graduate School of Frontier Sciences, The University of Tokyo(前沿科学研究生院,东京大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);Department of Biostatistics and Bioinformatics, Duke University(生物统计学与生物信息学系,杜克大学);Perelman School of Medicine, University of Pennsylvania(佩雷尔曼医学院,宾夕法尼亚大学);School of Clinical Medicine, University of Cambridge(临床医学学院,剑桥大学);Hospital of the University of Pennsylvania(宾夕法尼亚大学医院);Children’s Hospital of Philadelphia (CHOP)(费城儿童医院);Department of Anatomic Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania(解剖病理学与检验医学系,宾夕法尼亚大学医院);Department of Pathology and Laboratory Medicine, University of California, San Francisco(病理学与检验医学系,加州大学旧金山分校);Language Technologies Institute, Carnegie Mellon University(语言技术研究所,卡内基梅隆大学)
MIND: Empowering Mental Health Clinicians with Multimodal Data Insights through a Narrative Dashboard
Columbia University(哥伦比亚大学);University of Washington(华盛顿大学);Hamilton-Madison House(汉密尔顿-麦迪逊之家);New York-Presbyterian Hospital/ Weill Cornell Medicine(纽约长老会医院/威尔康奈尔医学);University of Oregon(俄勒冈大学);Cornell University(康奈尔大学);Northeastern University(东北大学)