Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks
Bytedance（字节跳动）；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；Mohamed bin Zayed University of Artificial Intelligence（马尔代夫人工智能大学）；Stanford University（斯坦福大学）；University of Michigan（密歇根大学）
A recent advancement in Multimodal Large Language Models (MLLMs) research is the emergence of "reasoning MLLMs" that offer explicit control over their internal thinking processes (normally referred as the "thinking mode") alongside the standard "non-thinking mode". This capability allows these models to engage in a step-by-step process of internal deliberation before generating a final response. With the rapid transition to and adoption of these "dual-state" MLLMs, this work rigorously evaluated how the enhanced reasoning processes of these MLLMs impact model performance and reliability in clinical tasks. This paper evaluates the active "thinking mode" capabilities of two leading MLLMs, Seed1.5-VL and Gemini-2.5-Flash, for medical applications. We assessed their performance on four visual medical tasks using VQA-RAD and ROCOv2 datasets. Our findings reveal that the improvement from activating the thinking mode remains marginal compared to the standard non-thinking mode for the majority of the tasks. Their performance on complex medical tasks such as open-ended VQA and medical image interpretation remains suboptimal, highlighting the need for domain-specific medical data and more advanced methods for medical knowledge integration.
CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Carnegie Mellon University（卡内基梅隆大学）；Hong Kong Baptist University（香港浸会大学）；University of Oxford（牛津大学）；New York University, Abu Dhabi（纽约大学阿布扎比分校）
Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Large Language Model Department, Tencent（腾讯大语言模型部）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong University of Science and Technology（香港科技大学）；Zhongguancun Academy（中关村学院）
Benchmarking Deep Learning-Based Reconstruction Methods for Photoacoustic Computed Tomography with Clinically Relevant Synthetic Datasets
Department of Bioengineering, University of Illinois Urbana-Champaign（生物工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign（电气与计算机工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Biomedical Engineering, The University of Texas at Austin（生物医学工程系，德克萨斯大学奥斯汀分校）；Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（奥登计算工程与科学研究所，德克萨斯大学奥斯汀分校）
Benchmarking Automated Clinical Language Simplification: Dataset, Algorithm, and Evaluation
College of Information Sciences and Technology, Pennsylvania State University（宾夕法尼亚州立大学信息科学与技术学院）；School of Software Technology, Dalian University of Technology（大连理工大学软件学院）；Relativity（相对论公司）
Benchmarking Clinical Decision Pathway Adherence in Large Language Models
Tongji University（同济大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tongji University School of Medicine（同济大学医学院）；Sun Yat-sen University（中山大学）；Fuzhou University（福州大学）；Shenzhen Loop Area Institute（深圳河套学院）
Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features
Department of Medical Informatics, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands（医学信息学系，阿姆斯特丹大学医学中心，阿姆斯特丹大学，阿姆斯特丹，荷兰）；Amsterdam Public Health, Methodology, Amsterdam, The Netherlands（阿姆斯特丹公共健康，方法学，阿姆斯特丹，荷兰）
Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates
University of Alberta（阿尔伯塔大学）；Stanford Center for Biomedical Informatics Research（斯坦福大学生物医学信息学研究中心）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Mass General Brigham（麻省总医院布里格姆）；Harvard Medical School（哈佛医学院）；Stanford University School of Medicine（斯坦福大学医学院）；Stanford Mussallem Center for Biodesign（斯坦福穆萨勒姆生物设计中心）；Stanford Health Care（斯坦福医疗保健）；University of British Columbia（不列颠哥伦比亚大学）；Stanford School of Medicine（斯坦福医学院）
DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
Princeton University（普林斯顿大学）；Boston University（波士顿大学）；New York University（纽约大学）；Flatiron Institute（Flatiron研究所）；Institute for Advanced Studies（高级研究研究所）
BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text
Brigham and Women's Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Massachusetts Institute of Technology（麻省理工学院）；Mayo Clinic（梅奥诊所）；Harvard T.H. Chan School of Public Health（哈佛大学陈曾熙公共卫生学院）；Harvard University（哈佛大学）；Stanford University（斯坦福大学）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Kempner Institute for the Study of Natural and Artificial Intelligence（肯普纳自然与人工智能研究所）；Broad Institute of MIT and Harvard（博德研究所）；Harvard Data Science Initiative（哈佛数据科学计划）
MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models
ETH Zurich（苏黎世联邦理工学院）；MBZUAI（马克斯·普朗克人工智能研究所）；Amazon（亚马逊）；University of Oxford（牛津大学）；University of Bern（伯尔尼大学）；Imperial College London（伦敦帝国理工学院）；Peking University（北京大学）
MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science
University of California, Los Angeles Computer Science Department（加州大学洛杉矶分校计算机科学系）；University of Pennsylvania Department of Materials Science and Engineering（宾夕法尼亚大学材料科学与工程系）；Virginia Tech Department of Computer Science（弗吉尼亚理工大学计算机科学系）
Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Renmin University of China（中国人民大学）；National Yang Ming Chiao Tung University；Taipei Veterans General Hospital（台北荣民总医院）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）
MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models
Harvard University（哈佛大学）；MIT（麻省理工学院）；Cornell University（康奈尔大学）；Mayo Clinic（梅奥诊所）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；University of Virginia（弗吉尼亚大学）；Harvard Medical School（哈佛医学院）；Abaka AI；NYU（纽约大学）
MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models
Seoul National University（首尔大学）；Seoul National University Hospital（首尔大学医院）；Seoul National University College of Medicine（首尔大学医学院）；Sungkyunkwan University School of Medicine（成均馆大学医学院）
Benchmarking Next-Generation Reasoning-Focused Large Language Models in Ophthalmology: A Head-to-Head Evaluation on 5,888 Items
National University of Singapore（新加坡国立大学）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore National Eye Centre（新加坡国家眼科中心）；The Chinese University of Hong Kong（香港中文大学）；Yale University（耶鲁大学）；Cleveland Clinic（克利夫兰诊所）；Montreal（蒙特利尔）；Tan Tock Seng Hospital（陈笃生医院）；Stanford University（斯坦福大学）
Dental-TriageBench: Benchmarking Multimodal Reasoning for Hierarchical Dental Triage
School of Computing and Data Science, The University of Hong Kong（香港大学计算与数据科学学院）；Faculty of Dentistry, The University of Hong Kong（香港大学牙科学院）；The Prince Philip Dental Hospital（菲利普王子牙科医院）；Li Ka Shing Faculty of Medicine, The University of Hong Kong（香港大学利滋医学学院）；The Hong Kong University of Science and Technology（香港科学与技术大学）
AMPLIFAI: A Multiphase CT Dataset for Benchmarking Clinical Reasoning in LI-RADS Assessment of Liver Lesions
University of Maryland, College Park（马里兰大学帕克分校）；University of Maryland School of Medicine（马里兰大学医学院）；University of Maryland Institute for Health Computing（马里兰大学健康计算研究所）；University of Maryland Medical System（马里兰大学医学系统）
Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks
Benchmarking and Adapting On-Device LLMs for Clinical Decision Support
AI Collaborative Centre, University Health Network（人工智能协同中心，健康大学网络）；Princess Margaret Cancer Centre, University Health Network（彭菲癌症中心，健康大学网络）；Department of Electrical and Computer Engineering, University of Toronto（电气与计算机工程系，多伦多大学）；Division of Urology, Department of Surgery, St. Michael’s Hospital, Unity Health Toronto and University of Toronto（泌尿科，外科部，圣米歇尔医院，统一健康多伦多及多伦多大学）；Peter Munk Cardiac Centre, University Health Network（彼得·默克心脏中心，健康大学网络）；Department of Laboratory Medicine and Pathobiology and Department of Computer Science, University of Toronto（实验室医学与病理学系及计算机科学系，多伦多大学）；Vector Institute, Toronto（向量研究所，多伦多）
RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Macau University of Science and Technology（澳门科技大学）；First Affiliated Hospital of Guangzhou Medical University（广州医科大学附属第一医院）；Guangzhou Institute of Respiratory Health（广州呼吸健康研究院）
