MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models
ETH Zurich(苏黎世联邦理工学院);MBZUAI(马克斯·普朗克人工智能研究所);Amazon(亚马逊);University of Oxford(牛津大学);University of Bern(伯尔尼大学);Imperial College London(伦敦帝国理工学院);Peking University(北京大学)
Boqi Chen, Xudong Liu, Jiachuan Peng, Marianne Frey-Marti, Bang Zheng, Kyle Lam, Lin Li, Jianing Qiu
Multimodal large language models (MLLMs) have shown great potential in medical applications, yet existing benchmarks inadequately capture real-world clinical complexity. We introduce MEDSYN, a multilingual, multimodal benchmark of highly complex clinical cases with up to 7 distinct visual clinical evidence (CE) types per case. Mirroring clinical workflow, we evaluate 18 MLLMs on differential diagnosis (DDx) generation and final diagnosis (FDx) selection. While top models often match or even outperform human experts on DDx generation, all MLLMs exhibit a much larger DDx--FDx performance gap compared to expert clinicians, indicating a failure mode in synthesis of heterogeneous CE types. Ablations attribute this failure to (i) overreliance on less discriminative textual CE ($\it{e.g.}$, medical history) and (ii) a cross-modal CE utilization gap. We introduce Evidence Sensitivity to quantify the latter and show that a smaller gap correlates with higher diagnostic accuracy. Finally, we demonstrate how it can be used to guide interventions to improve model performance. We will open-source our benchmark and code.
Benchmarking Clinical Decision Pathway Adherence in Large Language Models
Tongji University(同济大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tongji University School of Medicine(同济大学医学院);Sun Yat-sen University(中山大学);Fuzhou University(福州大学);Shenzhen Loop Area Institute(深圳河套学院)
Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features
Department of Medical Informatics, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands(医学信息学系,阿姆斯特丹大学医学中心,阿姆斯特丹大学,阿姆斯特丹,荷兰);Amsterdam Public Health, Methodology, Amsterdam, The Netherlands(阿姆斯特丹公共健康,方法学,阿姆斯特丹,荷兰)
MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models
Harvard University(哈佛大学);MIT(麻省理工学院);Cornell University(康奈尔大学);Mayo Clinic(梅奥诊所);UIUC(伊利诺伊大学厄巴纳-香槟分校);University of Virginia(弗吉尼亚大学);Harvard Medical School(哈佛医学院);Abaka AI;NYU(纽约大学)
RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Macau University of Science and Technology(澳门科技大学);First Affiliated Hospital of Guangzhou Medical University(广州医科大学附属第一医院);Guangzhou Institute of Respiratory Health(广州呼吸健康研究院)
Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction
The University of Texas at Austin(德克萨斯大学奥斯汀分校);New York University(纽约大学);The University of Texas Southwestern Medical Center(德克萨斯大学西南医学中心);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates
University of Alberta(阿尔伯塔大学);Stanford Center for Biomedical Informatics Research(斯坦福大学生物医学信息学研究中心);Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心);Mass General Brigham(麻省总医院布里格姆);Harvard Medical School(哈佛医学院);Stanford University School of Medicine(斯坦福大学医学院);Stanford Mussallem Center for Biodesign(斯坦福穆萨勒姆生物设计中心);Stanford Health Care(斯坦福医疗保健);University of British Columbia(不列颠哥伦比亚大学);Stanford School of Medicine(斯坦福医学院)
CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering
Department of Computer Science, University of Southern California(南加州大学计算机科学系);Department of Electrical and Computer Engineering, University of Southern California(南加州大学电气与计算机工程系);Suzanne Dworak-Peck School of Social Work, University of Southern California(南加州大学苏兹安·德沃拉克-佩克社会工作学院);Department of Psychiatry and the Behavioral Sciences, University of Southern California(南加州大学精神病学与行为科学系);Annenberg School for Communication, University of Southern California(南加州大学安纳伯格通信学院)
BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text
Brigham and Women's Hospital(布莱根妇女医院);Harvard Medical School(哈佛医学院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Massachusetts Institute of Technology(麻省理工学院);Mayo Clinic(梅奥诊所);Harvard T.H. Chan School of Public Health(哈佛大学陈曾熙公共卫生学院);Harvard University(哈佛大学);Stanford University(斯坦福大学);Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心);Kempner Institute for the Study of Natural and Artificial Intelligence(肯普纳自然与人工智能研究所);Broad Institute of MIT and Harvard(博德研究所);Harvard Data Science Initiative(哈佛数据科学计划)
Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench
The Chinese University of Hong Kong(香港中文大学);Tianjin Medical University(天津医科大学);Tencent AI Lab(腾讯AI实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Performance of large language models in numerical vs. semantic medical knowledge: Benchmarking on evidence-based Q&As
Tel-Aviv University(特拉维夫大学);Kahun Medical Ltd(Kahun医疗有限公司);Hebrew University of Jerusalem(耶路撒冷希伯来大学);Leumit Health Services(Leumit医疗服务机构);Bar-Ilan University(巴伊兰大学);Kaplan Medical Center(卡普兰医疗中心);Brigham and Women’s Hospital(布莱根妇女医院);Harvard Medical School(哈佛医学院)
Benchmarking Motivational Interviewing Competence of Large Language Models
National Institute of Mental Health and Neuro Sciences (NIMHANS)(国家精神卫生与神经科学研究所(NIMHANS));Indian Institute of Technology (IIT) Kharagpur(印度理工学院卡拉格普尔分校(IIT Kharagpur))
Benchmarking and Adapting On-Device LLMs for Clinical Decision Support
AI Collaborative Centre, University Health Network(人工智能协同中心,健康大学网络);Princess Margaret Cancer Centre, University Health Network(彭菲癌症中心,健康大学网络);Department of Electrical and Computer Engineering, University of Toronto(电气与计算机工程系,多伦多大学);Division of Urology, Department of Surgery, St. Michael’s Hospital, Unity Health Toronto and University of Toronto(泌尿科,外科部,圣米歇尔医院,统一健康多伦多及多伦多大学);Peter Munk Cardiac Centre, University Health Network(彼得·默克心脏中心,健康大学网络);Department of Laboratory Medicine and Pathobiology and Department of Computer Science, University of Toronto(实验室医学与病理学系及计算机科学系,多伦多大学);Vector Institute, Toronto(向量研究所,多伦多)
DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models
Shanghai Jiao Tong University(上海交通大学);SII;SPIRAL Lab(SPIRAL实验室);Generative AI Research Lab (GAIR)(生成式AI研究实验室);Shanghai Chest Hospital(上海胸科医院);Beijing Anzhen Hospital, Capital Medical University(北京安贞医院,首都医科大学)
Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);School of Artificial Intelligence, Shenzhen University(深圳大学人工智能学院);Guangdong Provincial Key Laboratory of Intelligent Information Processing(广东省智能信息处理重点实验室);The Hong Kong University of Science and Technology(香港科学与技术大学);Renmin University of China(中国人民大学);National Yang Ming Chiao Tung University;Taipei Veterans General Hospital(台北荣民总医院);School of Biomedical Engineering, Shenzhen University(深圳大学生物医学工程学院)
Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks
Bytedance(字节跳动);Peking University(北京大学);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);Mohamed bin Zayed University of Artificial Intelligence(马尔代夫人工智能大学);Stanford University(斯坦福大学);University of Michigan(密歇根大学)
HIVMedQA: Benchmarking large language models for HIV medical decision support
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland(1 生物系统科学与工程系,苏黎世联邦理工学院,巴塞尔,瑞士);School of Life Sciences, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland(2 生命科学学院,洛桑联邦理工学院,洛桑,瑞士);Swiss Institute of Bioinformatics, Lausanne, Switzerland(3 瑞士生物信息学研究所,洛桑,瑞士);Biomedical Data Science Center, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland(4 生物医学数据科学中心,洛桑大学医院和洛桑大学,洛桑,瑞士);Institute of Medical Virology, University of Zurich, Zurich, Switzerland(5 医学病毒学研究所,苏黎世大学,苏黎世,瑞士);Department of Infectious Diseases and Hospital Epidemiology, University Hospital Zurich, Zurich, Switzerland(6 感染病与医院流行病学系,苏黎世大学医院,苏黎世,瑞士);ETH AI Center, ETH Zurich, Zurich, Switzerland(7 ETH人工智能中心,苏黎世联邦理工学院,苏黎世,瑞士);Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland(8 量化生物医学系,苏黎世大学,苏黎世,瑞士)
PsychBench: A comprehensive and professional benchmark for evaluating the performance of LLM-assisted psychiatric clinical practice
Shanghai Jiao Tong University(上海交通大学);Beijing Anding Hospital, Capital Medical University(首都医科大学附属北京安定医院);Capital Medical University(首都医科大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Benchmarking ChatGPT-4 on ACR Radiation Oncology In-Training (TXIT) Exam and Red Journal Gray Zone Cases: Potentials and Challenges for AI-Assisted Medical Education and Decision Making in Radiation Oncology
University Hospital Erlangen(埃尔兰根大学医院);Comprehensive Cancer Center Erlangen-EMN(埃尔兰根-EMN综合癌症中心);Pattern Recognition Lab(模式识别实验室)
ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Centre for Medical Informatics, The University of Edinburgh(爱丁堡大学医学信息学中心);Health Data Research UK(英国健康数据研究所);School of Computer Science, Peking University(北京大学计算机学院);National Engineering Research Center for Software Engineering, Peking University(北京大学软件工程国家工程研究中心);ETH Zurich(苏黎世联邦理工学院);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院)
MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Yale University(耶鲁大学);Korea University(高丽大学);Beijing Normal University(北京师范大学);INSAIT, Sofia University, “St. Kliment Ohridski”(索非亚大学INSAIT研究所)
Benchmarking Automated Clinical Language Simplification: Dataset, Algorithm, and Evaluation
College of Information Sciences and Technology, Pennsylvania State University(宾夕法尼亚州立大学信息科学与技术学院);School of Software Technology, Dalian University of Technology(大连理工大学软件学院);Relativity(相对论公司)
Dental-TriageBench: Benchmarking Multimodal Reasoning for Hierarchical Dental Triage
School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院);Faculty of Dentistry, The University of Hong Kong(香港大学牙科学院);The Prince Philip Dental Hospital(菲利普王子牙科医院);Li Ka Shing Faculty of Medicine, The University of Hong Kong(香港大学利滋医学学院);The Hong Kong University of Science and Technology(香港科学与技术大学)
MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models
Seoul National University(首尔大学);Seoul National University Hospital(首尔大学医院);Seoul National University College of Medicine(首尔大学医学院);Sungkyunkwan University School of Medicine(成均馆大学医学院)