The application of the Multi-modal Large Language Models (MLLMs) in medical clinical scenarios remains underexplored. Previous benchmarks only focus on the capacity of the MLLMs in medical visual question-answering (VQA) or report generation and fail to assess the performance of the MLLMs on complex clinical multi-modal tasks. In this paper, we propose a novel Medical Personalized Multi-modal Consultation (Med-PMC) paradigm to evaluate the clinical capacity of the MLLMs. Med-PMC builds a simulated clinical environment where the MLLMs are required to interact with a patient simulator to complete the multi-modal information-gathering and decision-making task. Specifically, the patient simulator is decorated with personalized actors to simulate diverse patients in real scenarios. We conduct extensive experiments to access 12 types of MLLMs, providing a comprehensive view of the MLLMs' clinical performance. We found that current MLLMs fail to gather multimodal information and show potential bias in the decision-making task when consulted with the personalized patient simulators. Further analysis demonstrates the effectiveness of Med-PMC, showing the potential to guide the development of robust and reliable clinical MLLMs. Code and data are available at https://github.com/LiuHC0428/Med-PMC.
Beyond Captions: Context-Grounded Reconstruction for Biomedical Multimodal Continued Pretraining
The Hong Kong Polytechnic University（香港理工大学）；Sun Yat-sen University（中山大学）；InfiX.ai；PolyU-Daya Bay Technology and Innovation Research Institute（PolyU-大亚湾技术与创新研究院）
EMRModel: A Large Language Model for Extracting Medical Consultation Dialogues into Structured Medical Records
Taihe County People’s Hospital（太和县人民医院）；Innovation + Research Institute（创新+研究院）；GuoChuang Cloud Technology Ltd（国创云科技有限公司）；Xunfei Healthcare Technology Ltd（讯飞医疗科技有限公司）；University of Southern California（南加州大学）；The First Affiliated Hospital of Anhui Medical University（安徽医科大学第一附属医院）
DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation
School of Data Science, Fudan University（复旦大学数据科学学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；University of Toronto（多伦多大学）；School of Computer Science, Fudan University（复旦大学计算机学院）；Research Institute of Automatic and Complex Systems, Fudan University（复旦大学自动与复杂系统研究院）
Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；East China University of Science and Technology（华东理工大学）；SenseTime Research（商汤科技研究院）
Jingfang: An LLM-Based Multi-Agent System for Precise Medical Consultation and Syndrome Differentiation in Traditional Chinese Medicine
Beijing University of Posts and Telecommunications（北京邮电大学）；Southeast University（东南大学）；University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology of China（中国科学技术大学）
Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries: A Pilot Study
Faculty of Information Technology, Monash University（信息技术学院，莫纳什大学）；Faculty of Medicine, Nursing and Health Sciences, Monash University（医学、护理与健康科学学院，莫纳什大学）；Supportive and Palliative Care Unit, Monash Health（支持与姑息护理单元，莫纳什健康）
MedDG: An Entity-Centric Medical Consultation Dataset for Entity-Aware Medical Dialogue Generation
Sun Yat-sen University（中山大学）；Hong Kong University of Science and Technology（香港科技大学）；Hong Kong Polytechnic University（香港理工大学）；Tencent Jarvis Lab（腾讯天衍实验室）
Satisfactory Medical Consultation based on Terminology-Enhanced Information Retrieval and Emotional In-Context Learning
School of Mechanical Science and Engineering, Huazhong University of Science and Technology（华中科技大学机械科学与工程学院）；School of Computer Science, University of Warwick（华威大学计算机学院）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；MRC Weatherall Institute of Molecular Medicine, University of Oxford（牛津大学MRC Weatherall分子医学研究所）
LLMs for Medical Consultation Are Evaluated Too Late: The Preformulation Gap
Harvard T.H. Chan School of Public Health（哈佛陈曾熙公共卫生学院）；Harvard Medical School（哈佛医学院）；Australian Artificial Intelligence Institute, University of Technology Sydney（悉尼科技大学澳大利亚人工智能研究所）；Raycaster；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）
Rethinking Patient Education as Multi-turn Multi-modal Interaction
VA Bedford Health Care（VA贝德福德医疗中心）；UMass Amherst（马萨诸塞大学阿默斯特分校）；UMass Lowell（马萨诸塞大学洛厄尔分校）；Yale University（耶鲁大学）；National University of Singapore（新加坡国立大学）；Yale School of Medicine（耶鲁医学院）
From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with Biomedical Literature
Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, CT 06510, USA（耶鲁医学院生物医学信息学与数据科学系，耶鲁大学，新 Haven，CT 06510，USA）；School of Medicine, University of Puerto Rico, San Juan, PR 00921, USA（波多黎各大学医学院，波多黎各，San Juan，PR 00921，USA）
AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows
School of Control Science and Engineering, Shandong University（控制科学与工程学院，山东大学）；Key Laboratory of Machine Intelligence and System Control, Shandong University（机器智能与系统控制重点实验室，山东大学）；Department of Medicine and Therapeutics, The Chinese University of Hong Kong（医学与治疗学系，香港中文大学）；Department of Geriatric Medicine, Qilu Hospital of Shandong University（老年医学科，山东大学齐鲁医院）；Department of Psychiatry, The Chinese University of Hong Kong（精神病学系，香港中文大学）；Li Chiu Kong Family Sleep Assessment Unit, Department of Psychiatry, Faculty of Medicine, The Chinese University of Hong Kong（李秋虹家庭睡眠评估单元，精神病学系，医学院，香港中文大学）；Li Ka Shing Institute of Health Sciences, Faculty of Medicine, The Chinese University of Hong Kong（李嘉诚健康科学研究院，医学院，香港中文大学）；Gerald Choa Neuroscience Institute, Department of Medicine and Therapeutics, The Chinese University of Hong Kong（Gerald Choa 神经科学研究所，医学与治疗学系，香港中文大学）
Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates
University of Alberta（阿尔伯塔大学）；Stanford Center for Biomedical Informatics Research（斯坦福大学生物医学信息学研究中心）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Mass General Brigham（麻省总医院布里格姆）；Harvard Medical School（哈佛医学院）；Stanford University School of Medicine（斯坦福大学医学院）；Stanford Mussallem Center for Biodesign（斯坦福穆萨勒姆生物设计中心）；Stanford Health Care（斯坦福医疗保健）；University of British Columbia（不列颠哥伦比亚大学）；Stanford School of Medicine（斯坦福医学院）
PACT: Learning Diverse Diagnostic Strategies via Privileged Synthesis and Branch Consensus
Beihang University（北京航空航天大学）；Baidu（百度）；ByteDance（字节跳动）；Beijing Academy of Blockchain and Edge Computing（北京区块链与边缘计算研究院）；Renmin University of China（中国人民大学）
ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge
University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The Ohio State University（俄亥俄州立大学）
ColaCare: Enhancing Electronic Health Record Modeling through Large Language Model-Driven Multi-Agent Collaboration
Peking University（北京大学）；ETH Zurich（苏黎世联邦理工学院）；Peking University Third Hospital（北京大学第三医院）；University of Edinburgh（爱丁堡大学）；Beihang University（北京航空航天大学）；Health Data Research UK（英国健康数据研究中心）
