DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing
University of Wisconsin Madison（威斯康星大学麦迪逊分校）；Loyola University Chicago（芝加哥洛约拉大学）；Boston Children’s Hospital（波士顿儿童医院）；Harvard University（哈佛大学）；Rush University Medical Center（拉什大学医学中心）
Yanjun Gao, Dmitriy Dligach, Timothy Miller, John Caskey, Brihat Sharma, Matthew M Churpek, Majid Afshar
The meaningful use of electronic health records (EHR) continues to progress in the digital era with clinical decision support systems augmented by artificial intelligence. A priority in improving provider experience is to overcome information overload and reduce the cognitive burden so fewer medical errors and cognitive biases are introduced during patient care. One major type of medical error is diagnostic error due to systematic or predictable errors in judgment that rely on heuristics. The potential for clinical natural language processing (cNLP) to model diagnostic reasoning in humans with forward reasoning from data to diagnosis and potentially reduce the cognitive burden and medical error has not been investigated. Existing tasks to advance the science in cNLP have largely focused on information extraction and named entity recognition through classification tasks. We introduce a novel suite of tasks coined as Diagnostic Reasoning Benchmarks, DR.BENCH, as a new benchmark for developing and evaluating cNLP models with clinical diagnostic reasoning ability. The suite includes six tasks from ten publicly available datasets addressing clinical text understanding, medical knowledge reasoning, and diagnosis generation. DR.BENCH is the first clinical suite of tasks designed to be a natural language generation framework to evaluate pre-trained language models. Experiments with state-of-the-art pre-trained generative language models using large general domain models and models that were continually trained on a medical corpus demonstrate opportunities for improvement when evaluated in DR. BENCH. We share DR. BENCH as a publicly available GitLab repository with a systematic approach to load and evaluate models for the cNLP community.
Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation
GSAI, Renmin University of China（清华大学人工智能研究院，中国人民大学）；Tencent Jarvis Lab（腾讯 Jarvis实验室）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究关键实验室）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程技术研究中心，教育部）
Clinical Semantic Intelligence (CSI): Emulating the Cognitive Framework of the Expert Clinician for Comprehensive Oral Disease Diagnosis
Gilan University of Medical Sciences（加兹万大学医学科学学院）；University of Göttingen（哥廷根大学）；University of Tehran（德黑兰大学）；Hamadan University of Technology（哈马丹技术大学）；Islamic Azad University（伊斯兰 Azad 大学）
From Medical Records to Diagnostic Dialogues: A Clinical-Grounded Approach and Dataset for Psychiatric Comorbidity
X - LANCE Lab, Shanghai Jiao Tong University, China（上海交通大学X-LANCE实验室）；Shanghai Mental Health Center, SJTU School of Medicine, China（上海精神卫生中心，上海交通大学医学院）；Chen Frontier Lab for AI and Mental Health, Tianqiao and Chrissy Chen Institute, Shanghai, China（陈前沿人工智能与心理健康实验室，田青和克里斯蒂娜·陈研究所）
DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models
Premium Research Institute for Human Metaverse Medicine (WPI-PRIMe)（人类元宇宙医学高级研究机构（WPI-PRIMe））；Osaka University（大阪大学）；Department of Cardiology, The First Affiliated Hospital of Dalian Medical University（大连医科大学第一附属医院心内科）；Institute of Computing Technology, Chinese Academy of Science（中国科学院计算技术研究所）；D3 Center, Osaka University（大阪大学D3中心）；Agency for Science, Technology and Research (A*STAR)（科技研究局（A*STAR））
Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
Center for Biomedical Data Science, Duke-NUS Medical School（生物医学数据科学中心，杜克 - 新加坡国立大学医学院）；Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School（杜克 - 新加坡国立大学人工智能与医学科学计划，杜克 - 新加坡国立大学医学院）；Department of Population Health Sciences, Weill Cornell Medicine（人口健康科学系，威尔康奈尔医学院）；System Engineering, College of Engineering, Cornell University（系统工程，康奈尔大学工程学院）；Graduate School of Frontier Sciences, The University of Tokyo（前沿科学研究生院，东京大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Department of Biostatistics and Bioinformatics, Duke University（生物统计学与生物信息学系，杜克大学）；Perelman School of Medicine, University of Pennsylvania（佩雷尔曼医学院，宾夕法尼亚大学）；School of Clinical Medicine, University of Cambridge（临床医学学院，剑桥大学）；Hospital of the University of Pennsylvania（宾夕法尼亚大学医院）；Children’s Hospital of Philadelphia (CHOP)（费城儿童医院）；Department of Anatomic Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania（解剖病理学与检验医学系，宾夕法尼亚大学医院）；Department of Pathology and Laboratory Medicine, University of California, San Francisco（病理学与检验医学系，加州大学旧金山分校）；Language Technologies Institute, Carnegie Mellon University（语言技术研究所，卡内基梅隆大学）
A Structured Debate-Mixture-of-Agents Framework for Complex Clinical Diagnostic Decision Support
College of Computer Science, Sichuan University（四川大学计算机学院）；West China Hospital, Sichuan University（四川大学华西医院）；Med-X Center for Informatics, Sichuan University（四川大学华西医学信息学中心）
SemioLLM: Evaluating Large Language Models for Diagnostic Reasoning from Unstructured Clinical Narratives in Epilepsy
University of Tübingen（蒂宾根大学）；Technical University of Munich（慕尼黑工业大学）；University Clinic Tübingen（蒂宾根大学医院）；Hertie Institute for Clinical Brain Research（赫蒂临床脑研究所）；Excellence Cluster Machine Learning, Tübingen University（蒂宾根大学机器学习卓越集群）；Hertie Institute for AI in Brain Health (Hertie AI)（赫蒂人工智能脑健康研究所）
SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence
Beijing University of Technology（北京工业大学）；Beijing Institute of Technology（北京理工大学）；China-Japan Friendship Hospital（中日友好医院）；Dongbei University of Finance and Economics（东北财经大学）
MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs
National Chengchi University（国立中正大学）；Georgetown University（乔治城大学）；University of Michigan（密歇根大学）；Stevens Institute of Technology（史蒂文斯理工学院）；National Taiwan University of Science and Technology（台湾科技大学）；Far Eastern Memorial Hospital（东方纪念医院）
DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models
Shanghai Jiao Tong University（上海交通大学）；SII；SPIRAL Lab（SPIRAL实验室）；Generative AI Research Lab (GAIR)（生成式AI研究实验室）；Shanghai Chest Hospital（上海胸科医院）；Beijing Anzhen Hospital, Capital Medical University（北京安贞医院，首都医科大学）
Improving Interactive Diagnostic Ability of a Large Language Model Agent Through Clinical Experience Learning
Research Center of Computational Life Science, Zhejiang Lab（浙江实验室计算生命科学研究中心）；TransTek Medical（TransTek医疗）；Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University（浙江大学医学院附属邵逸夫医院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports
The D-Lab, Department of Precision Medicine, GROW - Research Institute for Oncology and Reproduction, Maastricht University（D实验室，精准医学系，GROW癌症与生殖研究研究所，马斯特里赫特大学）；Division of Cardiovascular Imaging, Department of Cardiology I, University Hospital Münster（心血管影像科，心内科第一部门，穆恩斯特大学医院）
Emulating Clinician Cognition via Self-Evolving Deep Clinical Research
Gaoling School of Artificial Intelligence（首都经济贸易大学人工智能学院）；Renmin University of China（中国人民大学）；Peking University Third Hospital（北京大学第三医院）；Baidu Inc.（百度公司）；Chinese PLA General Hospital（中国人民解放军总医院）；Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly（联合NTU-UBC老年积极生活方式卓越研究中心）；Nanyang Technological University（南洋理工大学）
Zodiac: A Cardiologist-Level LLM Framework for Multi-Agent Diagnostics
ZBeats Inc（ZBeats公司）；New York University（纽约大学）；Stony Brook Medicine（石溪医学中心）；University of Pennsylvania（宾夕法尼亚大学）；Binghamton University（宾汉姆顿大学）；Microsoft（微软公司）
Reverse Physician-AI Relationship: Full-process Clinical Diagnosis Driven by a Large Language Model
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University Third Hospital（北京大学第三医院）
Comparisons between a Large Language Model-based Real-Time Compound Diagnostic Medical AI Interface and Physicians for Common Internal Medicine Cases using Simulated Patients
Shihwa Medical Center（始华医疗中心）；Helpmedoc Inc.（Helpmedoc公司）；Asan Medical Center（峨山医疗中心）；University of Ulsan College of Medicine（蔚山大学医学院）
本研究开发了基于大语言模型的实时复合诊断医疗AI界面，通过非随机临床试验与医生比较，发现其在常见内科病例中诊断准确率更高（80% vs 50-70%）、时间更短（缩短44.6%）、成本更低（降低98.1%），患者满意度相当，显示出辅助初级保健的潜力。
Improving Clinical Diagnosis with Counterfactual Multi-Agent Reasoning
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Jacobi Medical Center, Albert Einstein College of Medicine（雅各布医疗中心，阿尔伯特·爱因斯坦学院）；Department of Emergency Medicine, Beth Israel Deaconess Medical Center（贝斯以色列医疗中心急诊科）；Leaning machines（Leanings machines）
A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics
The University of Hong Kong（香港大学）；West China Hospital, Sichuan University（四川大学华西医院）；Deepwise Healthcare（深睿医疗）；Zhuhai People’s Hospital and the First Affiliated Hospital of Faculty of Medicine, Macau University of Science and Technology and University Hospital（澳门科技大学医学院附属第一医院、珠海市人民医院及大学医院）；Peking University（北京大学）；Guangzhou Laboratory（广州实验室）；Jinling Hospital, Nanjing University School of Medicine（南京大学医学院附属金陵医院）
ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning
ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University（浙江大学杭州全球科技创新中心）；ZJU-UIUC Institute, Zhejiang University（浙江大学UIUC研究院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；The Polytechnic Institute, Zhejiang University（浙江大学 polytechnic 院）；AntGroup（蚂蚁集团）
Insight: A Multi-Modal Diagnostic Pipeline using LLMs for Ocular Surface Disease Diagnosis
Clinical Research Center, University of California, Berkeley（加州大学伯克利分校临床研究中心）；University of California, Berkeley（加州大学伯克利分校）；California Institute of Technology（加州理工学院）；University of California, Davis（加州大学戴维斯分校）；University of Hong Kong（香港大学）
