MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications
M42
Praveenkumar Kanithi, Clément Christophe, Marco AF Pimentel, Tathagata Raha, Prateek Munjal, Nada Saadi, Hamza A Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, Shadab Khan
While Large Language Models (LLMs) achieve superhuman performance on standardized medical licensing exams, these static benchmarks have become saturated and increasingly disconnected from the functional requirements of clinical workflows. To bridge the gap between theoretical capability and verified utility, we introduce MEDIC, a comprehensive evaluation framework establishing leading indicators of clinical LLM competence across five dimensions. These upfront indicators reveal cross-benchmark capability gaps, such as the divergence between static knowledge retrieval and functional execution, that inform model selection before costly deployment-based evaluation. Beyond standard question-answering, we assess operational capabilities using deterministic execution protocols and a novel Cross-Examination Framework (CEF), which quantifies information fidelity and hallucination rates without reliance on reference texts. Our evaluation across a heterogeneous task suite exposes critical performance trade-offs: we identify a significant knowledge-execution gap, where proficiency in static retrieval does not predict success in operational tasks such as clinical calculation or SQL generation. Furthermore, we observe a divergence between passive safety (refusal) and active safety (error detection), revealing that models fine-tuned for high refusal rates often fail to reliably audit clinical documentation for factual accuracy. These findings demonstrate that no single architecture dominates across all dimensions, highlighting the necessity of a portfolio approach to clinical model deployment. We accompany this work with a publicly available MEDIC leaderboard at https://hf.co/spaces/m42-health/MEDIC-Benchmark.
GraphMed-LT: Patient-Specific Graph Memory with Latent Clinical Thought Refinement for Multi-Turn Medical Conversations
School of Computing Science, University of Glasgow, UK(格拉斯哥大学计算机科学学院);School of Natural and Computing Science, University of Aberdeen, UK(阿伯丁大学自然与计算科学学院)
PrinciplismQA: A Philosophy-Grounded Approach to Assessing LLM-Human Clinical Medical Ethics Alignment
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));National Health Data Institute, Shenzhen(深圳国家健康数据研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院)
KREL: Automatic Medical Coding via Knowledge-Guided Reasoning over Clinical Evidence with LLMs
The University of New South Wales(新南威尔士大学);University of Göttingen(哥廷根大学);Macquarie University(麦考瑞大学);Beijing Intelligent Decision Medical Technology Co. Ltd(北京智决医疗科技有限公司)
Technical Report: Small Language Model for Japanese Clinical and Medicine
National Cerebral and Cardiovascular Center Hospital(国立循环器病研究中心医院);Graduate School of Medicine and Faculty of Medicine, Kyoto University(京都大学大学院医学研究科及医学部)
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
College of Computer Science and Artificial Intelligence, Fudan University(计算机科学与人工智能学院,复旦大学);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室);Shanghai AI Laboratory(上海人工智能实验室);Northwestern University(西北大学)
Medical Reasoning with Large Language Models: A Survey and MR-Bench
University of Science and Technology of China(中国科学技术大学);The First Affiliated Hospital of USTC(中国科学技术大学附属第一医院);National University of Singapore(新加坡国立大学)
Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding
Vector Institute for Artificial Intelligence(矢量人工智能研究所);University of Toronto(多伦多大学);Peter Munk Cardiac Centre, University Health Network(大学健康网络彼得·蒙克心脏中心);McGill University(麦吉尔大学);University Health Network(大学健康网络)
Development and Testing of a Novel Large Language Model-Based Clinical Decision Support Systems for Medication Safety in 12 Clinical Specialties
Singapore General Hospital(新加坡中央医院);Duke-NUS Medical School(杜克-新加坡国立大学医学院);Singapore National Eye Centre(新加坡全国眼科中心);Singapore Eye Research Institute(新加坡眼科研究所);Singapore Health Services(新加坡保健服务)
Humans and Large Language Models in Clinical Decision Support: A Study with Medical Calculators
National Library of Medicine (NLM), National Institutes of Health (NIH)(美国国立卫生研究院国立医学图书馆);University of Virginia(弗吉尼亚大学);Yale School of Medicine(耶鲁医学院);Massachusetts Eye and Ear, Harvard Medical School(哈佛医学院麻省眼耳医院);Boston Medical Center(波士顿医疗中心)
Automated Construction of Medical Indicator Knowledge Graphs Using Retrieval Augmented Large Language Models
The second hospital of Jilin University Northeast Asia Active Aging Laboratory Jilin, China Email;QMUL DERI \ Institute Of Health Informatics, Lonodn, UK Email;QMUL School of Electronic Engineering \ Computer Science London, UK Email;The second hospital of Jilin University College of Artificial Intelligence, \ University, Jilin, China Email
Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning
The Hong Kong Polytechnic University(香港理工大学);Hong Kong University(香港大学);South China University of Technology(华南理工大学);University of Toronto(多伦多大学);Peking Union Medical College Hospital(北京地坛医院);West China Hospital, Sichuan University(四川大学华西医院)
Utilizing Large Language Models for Zero-Shot Medical Ontology Extension from Clinical Notes
Department of Computer Science(计算机科学系);Emory University(埃默里大学);College of Public Health(公共卫生学院);Temple University(temple大学);School of Information(信息学院);Florida State University(佛罗里达州立大学);Hubert Department of Global Health(全球健康部);Nell Hodgson Woodruff School of Nursing(Nell Hodgson Woodruff护理学院)
Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions
School of Biomedical Engineering, Tsinghua University(清华大学生物医学工程学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Hepato-Pancreato-Biliary Center, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua University(清华大学临床医学院北京清华长庚医院肝胆胰中心);School of Computer and Communication Engineering, University of Science and Technology Beijing(北京科技大学计算机与通信工程学院);College of Electronic Information Engineering, Tongji University(同济大学电子与信息工程学院)
MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs
Seoul National University College of Medicine(首尔国立大学医学院);Seoul National University(首尔国立大学);Seoul National University College of Medicine, Seoul National University Hospital(首尔国立大学医学院、首尔国立大学医院)
Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks
NYU Langone Health(纽约大学朗格尼医学中心);The University of Texas at Austin(得克萨斯大学奥斯汀分校);Washington University School of Medicine in St. Louis(圣路易斯华盛顿大学医学院);New York University(纽约大学)