ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Centre for Medical Informatics, The University of Edinburgh（爱丁堡大学医学信息学中心）；Health Data Research UK（英国健康数据研究所）；School of Computer Science, Peking University（北京大学计算机学院）；National Engineering Research Center for Software Engineering, Peking University（北京大学软件工程国家工程研究中心）；ETH Zurich（苏黎世联邦理工学院）；School of Computing and Data Science, The University of Hong Kong（香港大学计算与数据科学学院）
Yinghao Zhu, Junyi Gao, Zixiang Wang, Weibin Liao, Xiaochen Zheng, Lifang Liang, Miguel O. Bernabeu, Yasha Wang, Lequan Yu, Chengwei Pan, Ewen M. Harrison, Liantao Ma
Large Language Models (LLMs) are increasingly deployed in medicine. However, their utility in non-generative clinical prediction, often presumed inferior to specialized models, remains under-evaluated, leading to ongoing debate within the field and potential for misuse, misunderstanding, or over-reliance due to a lack of systematic benchmarking. Our ClinicRealm study addresses this by benchmarking 15 GPT-style LLMs, 5 BERT-style models, and 11 traditional methods on unstructured clinical notes and structured Electronic Health Records (EHR), while also assessing their reasoning, reliability, and fairness. Key findings reveal a significant shift: for clinical note predictions, leading LLMs (e.g., DeepSeek-V3.1-Think, GPT-5) in zero-shot settings now decisively outperform finetuned BERT models. On structured EHRs, while specialized models excel with ample data, advanced LLMs (e.g., GPT-5, DeepSeek-V3.1-Think) show potent zero-shot capabilities, often surpassing conventional models in data-scarce settings. Notably, leading open-source LLMs can match or exceed proprietary counterparts. These results provide compelling evidence that modern LLMs are competitive tools for non-generative clinical prediction, particularly with unstructured text and offering data-efficient structured data options, thus necessitating a re-evaluation of model selection strategies. This research should serve as an important insight for medical informaticists, AI developers, and clinical researchers, potentially prompting a reassessment of current assumptions and inspiring new approaches to LLM application in predictive healthcare.
University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；Emory University（埃默里大学）；University of Minnesota（明尼苏达大学）；University of Massachusetts, Lowell（马萨诸塞大学洛厄尔分校）；UMass Chan Medical School（UMass Chan医学学院）
K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations
University of Roehampton（罗汉普顿大学）；Kivira Health；University of Hertfordshire（赫特福德大学）；University of Surrey（萨里大学）；University of Bedfordshire（贝德福德大学）；Tavistock Relationships（塔维斯托克关系中心）；InsideOut
SemioLLM: Evaluating Large Language Models for Diagnostic Reasoning from Unstructured Clinical Narratives in Epilepsy
University of Tübingen（蒂宾根大学）；Technical University of Munich（慕尼黑工业大学）；University Clinic Tübingen（蒂宾根大学医院）；Hertie Institute for Clinical Brain Research（赫蒂临床脑研究所）；Excellence Cluster Machine Learning, Tübingen University（蒂宾根大学机器学习卓越集群）；Hertie Institute for AI in Brain Health (Hertie AI)（赫蒂人工智能脑健康研究所）
A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined
DRIVE-Health CDT（DRIVE-Health 博士训练中心）；Department of Biostatistics and Health Informatics（生物统计学与健康信息学系）；Institute of Psychiatry, Psychology and Neuroscience（精神病学、心理学与神经科学研究所）；King’s College London（伦敦国王学院）；Cleveland Clinic London（伦敦克利夫兰诊所）
Evaluating Clinical Competencies of Large Language Models with a General Practice Benchmark
The Sixth Affiliated Hospital of Sun Yat-sen University（中山大学第六附属医院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济发展实验室（深圳））；Xinyi People’s Hospital（新一人民医院）；The Fifth Affiliated Hospital of Sun Yat-sen University（中山大学第五附属医院）；School of Public Health of Sun Yat-sen University（中山大学公共卫生学院）
ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making
Kyoto University, Department of Biomedical Data Intelligence（京都大学生物医学数据智能系）；Kyoto University, Department of Cardiovascular Medicine（京都大学心血管医学系）；The University of Tokyo（东京大学）
SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence
Beijing University of Technology（北京工业大学）；Beijing Institute of Technology（北京理工大学）；China-Japan Friendship Hospital（中日友好医院）；Dongbei University of Finance and Economics（东北财经大学）
PsychBench: A comprehensive and professional benchmark for evaluating the performance of LLM-assisted psychiatric clinical practice
Shanghai Jiao Tong University（上海交通大学）；Beijing Anding Hospital, Capital Medical University（首都医科大学附属北京安定医院）；Capital Medical University（首都医科大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations
Microsoft Research Asia（微软亚洲研究院）；Hong Kong University of Science and Technology（香港科技大学）；Peking University（北京大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Loyola University Chicago（芝加哥洛约拉大学）；Boston Children’s Hospital Harvard Medical School（波士顿儿童医院哈佛医学院）；University of Colorado-Anschutz（科罗拉多大学安舒茨分校）
Evaluating Large Language Models on the Spanish Medical Intern Resident (MIR) Examination 2024/2025:A Comparative Analysis of Clinical Reasoning and Knowledge Application
Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs
Department of Computer Science and Engineering, College of Engineering, Qatar University（卡塔尔大学计算机科学与工程系）；College of Science and Engineering, Hamad Bin Khalifa University (HBKU)（哈马德·本·卡伊夫大学（HBKU）理学院）；Primary Health Care Corporation (PHCC)（初级卫生保健公司）；Birmingham City University（伯明翰城市大学）
ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis
Zhejiang University School of Medicine（浙江大学医学院）；The First Affiliated Hospital（第一附属医院）；Liangzhu Laboratory（良渚实验室）；Shandong Jianzhu University（山东建筑大学）；FUXI AI Lab（伏羲人工智能实验室）；Beijing Normal University（北京师范大学）；Southern University of Science and Technology（南方科技大学）；China University of Geosciences (Wuhan)（中国地质大学（武汉））；Hangzhou Baorun Biotechnology Co., Ltd.（杭州宝润生物科技有限公司）
ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases
The Hong Kong Polytechnic University（香港理工大学）；Shenzhen Institutes of Advanced Technology, CAS（深圳先进技术研究院）；Zhujiang Hospital, Southern Medical University（南方医科大学珠江医院）；The Key Laboratory of Biomedical Imaging Science and System, CAS（生物医学成像科学与系统重点实验室）
Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
Center for Biomedical Data Science, Duke-NUS Medical School（生物医学数据科学中心，杜克 - 新加坡国立大学医学院）；Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School（杜克 - 新加坡国立大学人工智能与医学科学计划，杜克 - 新加坡国立大学医学院）；Department of Population Health Sciences, Weill Cornell Medicine（人口健康科学系，威尔康奈尔医学院）；System Engineering, College of Engineering, Cornell University（系统工程，康奈尔大学工程学院）；Graduate School of Frontier Sciences, The University of Tokyo（前沿科学研究生院，东京大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Department of Biostatistics and Bioinformatics, Duke University（生物统计学与生物信息学系，杜克大学）；Perelman School of Medicine, University of Pennsylvania（佩雷尔曼医学院，宾夕法尼亚大学）；School of Clinical Medicine, University of Cambridge（临床医学学院，剑桥大学）；Hospital of the University of Pennsylvania（宾夕法尼亚大学医院）；Children’s Hospital of Philadelphia (CHOP)（费城儿童医院）；Department of Anatomic Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania（解剖病理学与检验医学系，宾夕法尼亚大学医院）；Department of Pathology and Laboratory Medicine, University of California, San Francisco（病理学与检验医学系，加州大学旧金山分校）；Language Technologies Institute, Carnegie Mellon University（语言技术研究所，卡内基梅隆大学）
Evaluating the Impact of Lab Test Results on Large Language Models Generated Differential Diagnoses from Clinical Case Vignettes
Florida State University（佛罗里达州立大学）；National Center for Biotechnology Information, National Library of Medicine（美国国家医学图书馆国家生物技术信息中心）；Emory University（埃默里大学）；Morsani College of Medicine, University of South Florida（南佛罗里达大学莫萨尼医学院）
Evaluating GPT-5 as a Multimodal Clinical Reasoner: A Landscape Commentary
Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine（放射肿瘤科，Winship癌症研究所，埃默里大学医学院）；Department of Biomedical Engineering, Georgia Institute of Technology（生物医学工程系，佐治亚理工学院）
AcuityBench: Evaluating Clinical Acuity Identification and Uncertainty Alignment
Department of Computer Science, Columbia University, New York, NY, USA（计算机科学系，哥伦比亚大学，纽约，纽约州，美国）；Department of Biomedical Informatics, Columbia University, New York, NY, USA（生物医学信息学系，哥伦比亚大学，纽约，纽约州，美国）；Department of Emergency Medicine, Columbia University Irving Medical Center, New York, NY, USA（急诊医学系，哥伦比亚大学伊文思医疗中心，纽约，纽约州，美国）
Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset
School of ICT and Industrial Management（信息通信技术与工业管理学院）；Metropolia University of Applied Sciences（Metropolia应用科学大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
Bridging the Knowledge-Action Gap by Evaluating LLMs in Dynamic Dental Clinical Scenarios
The Second Affiliated Hospital of Harbin Medical University（哈尔滨医科大学附属第二医院）；Peking University School and Hospital of Stomatology（北京大学口腔医学院（口腔医院））；Medlinker Intelligent and Digital Technology Co., Ltd（医联智数科技有限公司）；The University of Queensland（昆士兰大学）；Beijing Xicheng District Health Care Hospital for Mothers and Children（北京市西城区妇幼保健院）
KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations
Kyoto University（京都大学）；The Hong Kong University of Science and Technology（香港科技大学）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；The University of Tokyo（东京大学）
SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Center for Language and Information Research, Wuhan University（武汉大学语言与信息研究中心）；Southwest Jiaotong University（西南交通大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；The University of Manchester（曼彻斯特大学）；Zhongnan Hospital of Wuhan University（武汉大学中南医院）
LLAMADRS: Evaluating Open-Source LLMs on Real Clinical Interviews--To Reason or Not to Reason?
Carnegie Mellon University, School of Computer Science（卡内基梅隆大学计算机学院）；University of Kansas, Department of Psychology（堪萨斯大学心理学系）；McLean Hospital, Harvard Medical School（麦肯纳医院哈佛医学院）
TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots
Department of Computer Science, Stanford University, Stanford, CA, USA（计算机科学系，斯坦福大学，斯坦福，加州，美国）；Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA（精神病学与行为科学系，斯坦福大学，斯坦福，加州，美国）；Department of Biomedical Data Science, Stanford University, Stanford, CA, USA（生物医学数据科学系，斯坦福大学，斯坦福，加州，美国）；University of California, Berkeley, Berkeley, CA, USA（加州大学伯克利分校，伯克利，加州，美国）；The University of Hong Kong, Hong Kong（香港大学，香港）
