Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?
Wits MIND Institute, University of the Witwatersrand, Johannesburg, South Africa(维特士心理研究所,沃斯兰德大学,约翰内斯堡,南非);Grai Labs, Cape Town, South Africa(格雷实验室,开普敦,南非);South African Medical Research Council Vaccines and Infectious Diseases Analytics Research Unit, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa(南非医学研究理事会疫苗和传染病分析研究组,健康科学学院,沃斯兰德大学,约翰内斯堡,南非);Department of Internal Medicine, Charlotte Maxeke Johannesburg Academic Hospital, and Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa(内科学系,查理·马克斯凯约翰内斯堡学术医院,以及健康科学学院,沃斯兰德大学,约翰内斯堡,南非);Department of Paediatrics and Child Health, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa(儿科学与儿童健康系,健康科学学院,沃斯兰德大学,约翰内斯堡,南非);Wits MIND Institute, University of the Witwatersrand, Johannesbu(维特士心理研究所,沃斯兰德大学,约翰内斯堡)
Amy Rouillard, Sitwala Mundia, Linda Camara, Ziyaad Dangor, Michael Cameron Gramanie, Ismail Kalla, Shabir A. Madhi, Kajal Morar, Marlvin T. Ncube, Haroon Saloo… 展开作者
Amy Rouillard, Sitwala Mundia, Linda Camara, Ziyaad Dangor, Michael Cameron Gramanie, Ismail Kalla, Shabir A. Madhi, Kajal Morar, Marlvin T. Ncube, Haroon Saloojee, Bruce A. Bassett
备注:
Extended version of a paper that appeared in Artificial Intelligence in Medicine 2026
Evaluating medical AI systems using expert clinician panels is costly and slow, motivating the use of large language models (LLMs) as alternative adjudicators. Here, we evaluate an LLM Jury, composed of three frontier AI models, for scoring 3334 diagnoses on 300 real-world low- and middle-income country (LMIC) hospital cases. Both LLM- and clinician-generated diagnoses are scored against expert panel diagnoses across four dimensions: diagnosis, differential diagnosis, clinical reasoning, and negative treatment risk. The LLM Jury scores are compared with expert and independent re-scoring panel scores to assess error metrics, inter-rater agreement, severe-risk errors, and the effect of post hoc calibration using isotonic regression. In our data, we find that: (i) the uncalibrated LLM Jury scores preserve ordinal agreement with the expert clinician panel scores, but are systematically lower; (ii) the probability of severe-risk errors is lower for the LLM Jury than the human expert re-score panels; (iii) the LLM Jury combined with LLM diagnoses can be used to identify diagnoses at high risk of error, enabling targeted expert review and improved panel efficiency; (iv) the calibrated LLM Jury scores and rankings of diagnosing agents show excellent agreement with those of the primary expert panels; (v) LLM Jury models show no self-preference bias, they did not score diagnoses generated by their own underlying model or models from the same vendor more (or less) favourably than those generated by other models. Together, these results provide evidence that a calibrated LLM Jury is a trustworthy and reliable proxy for expert clinician evaluation in medical AI benchmarking. Confirming these findings in other clinical settings is an important direction for future work.
Ask Before You Diagnose: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry
National University of Science and Technology Politehnica Bucharest(布加勒斯特理工大学国立科技大学);Psychiatric Hospital Doctor Gheorghe Preda(格奥尔基·普雷达医生精神病医院);Oslo Metropolitan University(奥斯陆都市大学);Kristiania University of Applied Sciences(克里斯蒂亚尼亚应用科学大学);SimulaMet(SimulaMet公司);Lucian Blaga University of Sibiu(锡比乌卢西安·布拉加大学)
Evaluating the Impact of Lab Test Results on Large Language Models Generated Differential Diagnoses from Clinical Case Vignettes
Florida State University(佛罗里达州立大学);National Center for Biotechnology Information, National Library of Medicine(美国国家医学图书馆国家生物技术信息中心);Emory University(埃默里大学);Morsani College of Medicine, University of South Florida(南佛罗里达大学莫萨尼医学院)
Human-AI collectives produce the most accurate differential diagnoses
Max Planck Institute for Human Development(马克斯·普朗克人类发展研究所);The Human Diagnosis Project(人类诊断项目);University of Cologne(科隆大学);Harvey Mudd College(哈维穆德学院);Oxford University(牛津大学);Kaiser Permanente(凯撒永久医疗);ISTC-CNR(意大利国家研究委员会认知科学与技术研究所);Technical University Berlin(柏林工业大学)
Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?
Carnegie Mellon University(卡内基梅隆大学);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Australian National University(澳大利亚国立大学);University of Western Australia(西澳大学);POSTECH;Yale University(耶鲁大学)
GraphMed-LT: Patient-Specific Graph Memory with Latent Clinical Thought Refinement for Multi-Turn Medical Conversations
School of Computing Science, University of Glasgow, UK(格拉斯哥大学计算机科学学院);School of Natural and Computing Science, University of Aberdeen, UK(阿伯丁大学自然与计算科学学院)
Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning
The Hong Kong Polytechnic University(香港理工大学);Hong Kong University(香港大学);South China University of Technology(华南理工大学);University of Toronto(多伦多大学);Peking Union Medical College Hospital(北京地坛医院);West China Hospital, Sichuan University(四川大学华西医院)
Utilizing Large Language Models for Zero-Shot Medical Ontology Extension from Clinical Notes
Department of Computer Science(计算机科学系);Emory University(埃默里大学);College of Public Health(公共卫生学院);Temple University(temple大学);School of Information(信息学院);Florida State University(佛罗里达州立大学);Hubert Department of Global Health(全球健康部);Nell Hodgson Woodruff School of Nursing(Nell Hodgson Woodruff护理学院)
PrinciplismQA: A Philosophy-Grounded Approach to Assessing LLM-Human Clinical Medical Ethics Alignment
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));National Health Data Institute, Shenzhen(深圳国家健康数据研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院)
KREL: Automatic Medical Coding via Knowledge-Guided Reasoning over Clinical Evidence with LLMs
The University of New South Wales(新南威尔士大学);University of Göttingen(哥廷根大学);Macquarie University(麦考瑞大学);Beijing Intelligent Decision Medical Technology Co. Ltd(北京智决医疗科技有限公司)
Leveraging Large Language Models to Extract Information on Substance Use Disorder Severity from Clinical Notes: A Zero-shot Learning Approach
Oak Ridge National Laboratory(橡树岭国家实验室);Program Evaluation and Resource Center, Office of Mental Health and Suicide Prevention, Department of Veterans Affairs(退伍军人事务部精神健康与自杀预防办公室项目评估与资源中心)
Medical Reasoning with Large Language Models: A Survey and MR-Bench
University of Science and Technology of China(中国科学技术大学);The First Affiliated Hospital of USTC(中国科学技术大学附属第一医院);National University of Singapore(新加坡国立大学)
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
College of Computer Science and Artificial Intelligence, Fudan University(计算机科学与人工智能学院,复旦大学);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室);Shanghai AI Laboratory(上海人工智能实验室);Northwestern University(西北大学)
Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding
Vector Institute for Artificial Intelligence(矢量人工智能研究所);University of Toronto(多伦多大学);Peter Munk Cardiac Centre, University Health Network(大学健康网络彼得·蒙克心脏中心);McGill University(麦吉尔大学);University Health Network(大学健康网络)
Humans and Large Language Models in Clinical Decision Support: A Study with Medical Calculators
National Library of Medicine (NLM), National Institutes of Health (NIH)(美国国立卫生研究院国立医学图书馆);University of Virginia(弗吉尼亚大学);Yale School of Medicine(耶鲁医学院);Massachusetts Eye and Ear, Harvard Medical School(哈佛医学院麻省眼耳医院);Boston Medical Center(波士顿医疗中心)
A Comprehensive Study on Machine Learning Methods to Increase the Prediction Accuracy of Classifiers and Reduce the Number of Medical Tests Required to Diagnose Alzheimer'S Disease
Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis
Penn State University(宾夕法尼亚州立大学);Mississippi State University(密西西比州立大学);Northwest Missouri State University(西北密苏里州立大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of California, Berkeley(加利福尼亚大学伯克利分校);Colorado State University(科罗拉多州立大学)