Current benchmarks for evaluating large language models (LLMs) in medical calculation are largely based on simplified settings, where each patient case corresponds to a single calculator and the required tool is explicitly specified in the query. However, real clinical scenarios often require multiple calculators for joint evaluation, nested-scale calculation, and fuzzy queries that do not directly specify the target calculator. To this end, we propose a new medical calculation benchmark, MedCalc-Pro, which covers three progressively challenging task settings: single-calculator, multi-calculator, and nested-calculator calculation settings. MedCalc-Pro contains 2,268 real-world clinical cases, covering 77 medical calculators across 14 clinical departments. Meanwhile, to address the limited performance of existing frameworks and methods in complex clinical scenarios, we further propose a more generalizable agent framework that supports multi-tool selection and nested-tool calling, while suppressing parameter error propagation through structured validation and evidence review. We conduct systematic comparisons across open-source, closed-source, and medical-specialized LLMs, and the results show that our framework achieves the best performance across all three task settings. This work provides a new benchmark and method for evaluating and applying LLMs in challenging medical calculation scenarios.
MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
National Library of Medicine, National Institutes of Health(美国国立卫生研究院国家医学图书馆);University of Virginia(弗吉尼亚大学);University of Illinois at Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校);Lapis Labs(拉皮斯实验室);Yale University School of Medicine(耶鲁大学医学院);Rosalind Franklin University Chicago Medical School(罗莎琳德·富兰克林大学芝加哥医学院);Howard University College of Medicine(霍华德大学医学院);University of Illinois College of Medicine at Chicago(芝加哥伊利诺伊大学医学院);University of Chicago Pritzker School of Medicine(芝加哥大学普利兹克医学院)
From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations
Department of Computer Science, Yale University, CT, USA(耶鲁大学计算机科学系);Center for Healthcare Organization and Implementation Research, VA Bedford Health Care(VA贝福德医疗中心健康组织与实施研究中心);Miner School of Computer and Information Sciences, UMass Lowell, MA, USA(UMass洛厄尔矿尔计算机与信息科学学院);Manning College of Information and Computer Sciences, UMass Amherst, MA, USA(UMass阿默斯特马宁信息与计算机科学学院)
RiskAgent: Synergizing Language Models with Validated Tools for Evidence-Based Risk Prediction
University of Oxford(牛津大学);University College London(伦敦大学学院);Technical University of Munich(慕尼黑技术大学);University of Glasgow(格拉斯哥大学);Oxford-Suzhou Centre for Advanced Research(牛津-苏浙高级研究中心)
Scalable Stewardship of an LLM-Assisted Clinical Benchmark with Physician Oversight
Department of Operations, Information and Technology, Stanford Graduate School of Business, Stanford, CA, USA(斯坦福大学商学院运营、信息与技术系);Department of Pediatrics, Division of Critical Care Medicine, Stanford University School of Medicine, Stanford, CA, USA(斯坦福大学医学院儿科系重症监护医学部);Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, Stanford, CA, USA(斯坦福大学医学院麻醉学、围手术期与疼痛医学系);Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA(斯坦福大学医学院放射肿瘤学系);Department of Electrical Engineering, Stanford University School of Engineering, Stanford, CA, USA(斯坦福大学工程学院电气工程系)
EMRModel: A Large Language Model for Extracting Medical Consultation Dialogues into Structured Medical Records
Taihe County People’s Hospital(太和县人民医院);Innovation + Research Institute(创新+研究院);GuoChuang Cloud Technology Ltd(国创云科技有限公司);Xunfei Healthcare Technology Ltd(讯飞医疗科技有限公司);University of Southern California(南加州大学);The First Affiliated Hospital of Anhui Medical University(安徽医科大学第一附属医院)
Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Ray
The Chinese University of Hong Kong(香港中文大学);CU Lab for AI in Radiology (CLAIR), The Chinese University of Hong Kong(香港中文大学放射学人工智能实验室(CLAIR));Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);China Unicom Global Limited(中国联通国际有限公司)
Medical Multimodal Model Stealing Attacks via Adversarial Domain Alignment
Bosch Center for Artificial Intelligence(博世人工智能中心);Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Saarland University(萨尔大学);University of Strasbourg(斯特拉斯堡大学);IHU Strasbourg(斯特拉斯堡大学医院研究所);CISPA Helmholtz Center for Information Security(CISPA亥姆霍兹信息安全中心)
MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making
Massachusetts Institute of Technology(麻省理工学院);Yonsei University College of Medicine(延世大学医学院);Google Research(谷歌研究院);Seoul National University Hospital(首尔大学医院)
The Forgotten Shield: Safety Grafting in Parameter-Space for Medical MLLMs
National University of Defense Technology(国防科技大学);Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统(MAIS));Multimedia Department, Xiaomi Inc(小米公司多媒体部门);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong(香港科学院人工智能与机器人中心);School of Artificial Intelligence, University of Chinese Academy of Sciences, UCAS(中国科学院大学人工智能学院)
Readying Medical Students for Medical AI: The Need to Embed AI Ethics Education
Applied Artificial Intelligence Institute, Deakin University(迪肯大学应用人工智能研究所);Centre for AI and Digital Ethics, School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院人工智能与数字伦理中心)