Medical conversational AI (AI) plays a pivotal role in the development of safer and more effective medical dialogue systems. However, existing benchmarks and evaluation frameworks for assessing the information-gathering and diagnostic reasoning abilities of medical large language models (LLMs) have not been rigorously evaluated. To address these gaps, we present MedDialogRubrics, a novel benchmark comprising 5,200 synthetically constructed patient cases and over 60,000 fine-grained evaluation rubrics generated by LLMs and subsequently refined by clinical experts, specifically designed to assess the multi-turn diagnostic capabilities of LLM. Our framework employs a multi-agent system to synthesize realistic patient records and chief complaints from underlying disease knowledge without accessing real-world electronic health records, thereby mitigating privacy and data-governance concerns. We design a robust Patient Agent that is limited to a set of atomic medical facts and augmented with a dynamic guidance mechanism that continuously detects and corrects hallucinations throughout the dialogue, ensuring internal coherence and clinical plausibility of the simulated cases. Furthermore, we propose a structured LLM-based and expert-annotated rubric-generation pipeline that retrieves Evidence-Based Medicine (EBM) guidelines and utilizes the reject sampling to derive a prioritized set of rubric items ("must-ask" items) for each case. We perform a comprehensive evaluation of state-of-the-art models and demonstrate that, across multiple assessment dimensions, current models face substantial challenges. Our results indicate that improving medical dialogue will require advances in dialogue management architectures, not just incremental tuning of the base-model.
EMRModel: A Large Language Model for Extracting Medical Consultation Dialogues into Structured Medical Records
Taihe County People’s Hospital(太和县人民医院);Innovation + Research Institute(创新+研究院);GuoChuang Cloud Technology Ltd(国创云科技有限公司);Xunfei Healthcare Technology Ltd(讯飞医疗科技有限公司);University of Southern California(南加州大学);The First Affiliated Hospital of Anhui Medical University(安徽医科大学第一附属医院)
DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation
School of Data Science, Fudan University(复旦大学数据科学学院);School of Computer Science, Northwestern Polytechnical University(西北工业大学计算机学院);University of Toronto(多伦多大学);School of Computer Science, Fudan University(复旦大学计算机学院);Research Institute of Automatic and Complex Systems, Fudan University(复旦大学自动与复杂系统研究院)
Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries: A Pilot Study
Faculty of Information Technology, Monash University(信息技术学院,莫纳什大学);Faculty of Medicine, Nursing and Health Sciences, Monash University(医学、护理与健康科学学院,莫纳什大学);Supportive and Palliative Care Unit, Monash Health(支持与姑息护理单元,莫纳什健康)
Jingfang: An LLM-Based Multi-Agent System for Precise Medical Consultation and Syndrome Differentiation in Traditional Chinese Medicine
Beijing University of Posts and Telecommunications(北京邮电大学);Southeast University(东南大学);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学)
Satisfactory Medical Consultation based on Terminology-Enhanced Information Retrieval and Emotional In-Context Learning
School of Mechanical Science and Engineering, Huazhong University of Science and Technology(华中科技大学机械科学与工程学院);School of Computer Science, University of Warwick(华威大学计算机学院);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);MRC Weatherall Institute of Molecular Medicine, University of Oxford(牛津大学MRC Weatherall分子医学研究所)
LLMs for Medical Consultation Are Evaluated Too Late: The Preformulation Gap
Harvard T.H. Chan School of Public Health(哈佛陈曾熙公共卫生学院);Harvard Medical School(哈佛医学院);Australian Artificial Intelligence Institute, University of Technology Sydney(悉尼科技大学澳大利亚人工智能研究所);Raycaster;Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心)
Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);East China University of Science and Technology(华东理工大学);SenseTime Research(商汤科技研究院)
AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows
School of Control Science and Engineering, Shandong University(控制科学与工程学院,山东大学);Key Laboratory of Machine Intelligence and System Control, Shandong University(机器智能与系统控制重点实验室,山东大学);Department of Medicine and Therapeutics, The Chinese University of Hong Kong(医学与治疗学系,香港中文大学);Department of Geriatric Medicine, Qilu Hospital of Shandong University(老年医学科,山东大学齐鲁医院);Department of Psychiatry, The Chinese University of Hong Kong(精神病学系,香港中文大学);Li Chiu Kong Family Sleep Assessment Unit, Department of Psychiatry, Faculty of Medicine, The Chinese University of Hong Kong(李秋虹家庭睡眠评估单元,精神病学系,医学院,香港中文大学);Li Ka Shing Institute of Health Sciences, Faculty of Medicine, The Chinese University of Hong Kong(李嘉诚健康科学研究院,医学院,香港中文大学);Gerald Choa Neuroscience Institute, Department of Medicine and Therapeutics, The Chinese University of Hong Kong(Gerald Choa 神经科学研究所,医学与治疗学系,香港中文大学)
PACT: Learning Diverse Diagnostic Strategies via Privileged Synthesis and Branch Consensus
Beihang University(北京航空航天大学);Baidu(百度);ByteDance(字节跳动);Beijing Academy of Blockchain and Edge Computing(北京区块链与边缘计算研究院);Renmin University of China(中国人民大学)
Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation
GSAI, Renmin University of China(清华大学人工智能研究院,中国人民大学);Tencent Jarvis Lab(腾讯 Jarvis实验室);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理研究关键实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程技术研究中心,教育部)
Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability
Leiden University Medical Centre(莱顿大学医学中心);The Leiden Institute of Advanced Computer Science(莱顿高级计算机科学研究所)
ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge
University of Texas Southwestern Medical Center(德克萨斯大学西南医学中心);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);The Ohio State University(俄亥俄州立大学)
A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations
School of Computer & Communication Engineering, University of Science and Technology Beijing(北京科技大学计算机与通信工程学院);School of Computer Science, University of South China(南华大学计算机学院);Faculty of Computer and Information Sciences, Hosei University(法政大学计算机与信息科学学院)
MedDG: An Entity-Centric Medical Consultation Dataset for Entity-Aware Medical Dialogue Generation
Sun Yat-sen University(中山大学);Hong Kong University of Science and Technology(香港科技大学);Hong Kong Polytechnic University(香港理工大学);Tencent Jarvis Lab(腾讯天衍实验室)
JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models
Kyoto University(京都大学);Hohai University(河海大学);The University of Tokyo(东京大学);University of Science and Technology of China(中国科学技术大学);Hong Kong Polytechnic University(香港理工大学)
ColaCare: Enhancing Electronic Health Record Modeling through Large Language Model-Driven Multi-Agent Collaboration
Peking University(北京大学);ETH Zurich(苏黎世联邦理工学院);Peking University Third Hospital(北京大学第三医院);University of Edinburgh(爱丁堡大学);Beihang University(北京航空航天大学);Health Data Research UK(英国健康数据研究中心)