Building effective clinical decision support systems requires the synthesis of complex heterogeneous multimodal data. Such modalities include temporal electronic health records data, medical images, radiology reports, and clinical notes. Large language model (LLM)-based agents have shown impressive performance in various healthcare tasks, especially those involving textual modalities. Considering the fragmentation of healthcare data across hospital systems, collaborative agent frameworks present a promising direction to mitigate data sharing challenges. However, the effectiveness of LLM agents for multimodal clinical risk prediction remains largely unexamined. In this work, we conduct a systematic evaluation of LLM-based agents for clinical prediction tasks using large-scale real-world data. We assess performance in unimodal and multimodal settings and quantify performance gaps between single agent and multi-agent systems. Our findings highlight that single agent frameworks outperform naive multi-agent systems, are better at handling multimodal data, and are better calibrated. This underscores a critical need for improving multi-agent collaboration to better handle heterogeneous inputs. By open-sourcing our code and evaluation framework, this work offers a new benchmark to support future developments relating to agentic systems in healthcare.
REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models
Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);School of Software, Beihang University(北京航空航天大学软件学院);China Mobile Research Institute(中国移动研究院)
Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions
School of Biomedical Engineering, Tsinghua University(清华大学生物医学工程学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Hepato-Pancreato-Biliary Center, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua University(清华大学临床医学院北京清华长庚医院肝胆胰中心);School of Computer and Communication Engineering, University of Science and Technology Beijing(北京科技大学计算机与通信工程学院);College of Electronic Information Engineering, Tongji University(同济大学电子与信息工程学院)
Large Language Models as Unified Multimodal Learners for Clinical Prediction
German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));Charité Universitätsmedizin Berlin(柏林夏里特大学医学中心);DNC Information Management GmbH(DNC信息管理有限公司);Klinik für Akut- und Notfallmedizin, Asklepios Klinikum Harburg(哈堡阿斯克勒庇俄斯医院急性与急诊医学科);Technical University Berlin(柏林工业大学)
Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Models
Hong Kong Baptist University(香港 Baptist 大学);Shanxi University(山西大学);Shanghai Institute for Advanced Study of Zhejiang University(浙江大学上海先进研究院);Hong Kong University of Science and Technology(香港科技大学);The University of Manchester(曼彻斯特大学)
OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks
Chinese University of Hong Kong(香港中文大学);Sun Yat-sen Memorial Hospital, Sun Yat-sen University(中山大学孙逸仙纪念医院);Southern Medical University(南方医科大学);Zhongshan Hospital, Fudan University(复旦大学中山医院);Department of Neurosurgery, Prince of Wales Hospital(威尔士王室医院神经外科部)
CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation
Center for Artificial Intelligence in Medicine and Imaging(医学与影像人工智能中心);Department of Radiology(放射科);Department of Pediatrics(儿科);Department of Radiology, Weill Cornell Medicine(韦氏 Cornell 医学院放射科);Department of Radiology, Brigham and Women’s Hospital(布里奇妇产科医院放射科);University of California, Berkeley(加州大学伯克利分校);University of California, San Francisco(加州大学旧金山分校);Department of Biomedical Data Science(生物医学数据科学部)
MedRLM: Recursive Multimodal Health Intelligence for Long-Context Clinical Reasoning, Sensor-Guided Screening, Evidence-Grounded Decision Support, and Community-to-Tertiary Referral Optimization
School of Information, Computer;Communication Technology Sirindhorn International Institute of Technology, Thammasat University Pathum Thani, Thailand 1
Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
Center for Biomedical Data Science, Duke-NUS Medical School(生物医学数据科学中心,杜克 - 新加坡国立大学医学院);Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School(杜克 - 新加坡国立大学人工智能与医学科学计划,杜克 - 新加坡国立大学医学院);Department of Population Health Sciences, Weill Cornell Medicine(人口健康科学系,威尔康奈尔医学院);System Engineering, College of Engineering, Cornell University(系统工程,康奈尔大学工程学院);Graduate School of Frontier Sciences, The University of Tokyo(前沿科学研究生院,东京大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);Department of Biostatistics and Bioinformatics, Duke University(生物统计学与生物信息学系,杜克大学);Perelman School of Medicine, University of Pennsylvania(佩雷尔曼医学院,宾夕法尼亚大学);School of Clinical Medicine, University of Cambridge(临床医学学院,剑桥大学);Hospital of the University of Pennsylvania(宾夕法尼亚大学医院);Children’s Hospital of Philadelphia (CHOP)(费城儿童医院);Department of Anatomic Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania(解剖病理学与检验医学系,宾夕法尼亚大学医院);Department of Pathology and Laboratory Medicine, University of California, San Francisco(病理学与检验医学系,加州大学旧金山分校);Language Technologies Institute, Carnegie Mellon University(语言技术研究所,卡内基梅隆大学)
DERM-3R: A Resource-Efficient Multimodal Agents Framework for Dermatologic Diagnosis and Treatment in Real-World Clinical Settings
School of Pharmacy, Nanjing University of Chinese Medicine(南京中医药大学药学院);Department of Dermatology, the Gulou Hospital of Traditional Chinese Medicine of Beijing(北京鼓楼中医医院皮肤科);Infectious disease department, Dongfang Hospital, Beijing University of Chinese Medicine(北京中医药大学东方医院感染科);College of Art and Science, university of Washington(华盛顿大学文理学院);Department of Computer Science, Johns Hopkins University(约翰霍普金斯大学计算机科学系);Fudan Unversity(复旦大学);Zhejiang Lab(之江实验室);University College London(伦敦大学学院);Department of Dermatology, Inner Mongolia Hospital of Traditional Chinese Medicine(内蒙古自治区中医医院皮肤科);International Campus of Zhejiang University(浙江大学国际校区)
Evaluating GPT-5 as a Multimodal Clinical Reasoner: A Landscape Commentary
Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine(放射肿瘤科,Winship癌症研究所,埃默里大学医学院);Department of Biomedical Engineering, Georgia Institute of Technology(生物医学工程系,佐治亚理工学院)
Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients
Faculty of Engineering, Shenzhen MSU-BIT University(深圳北理莫斯科大学工程学院);International School, Beijing University of Posts and Telecommunications(北京邮电大学国际学院);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);School of Artificial Intelligence and Computer Science, Jiangnan University(江南大学人工智能与计算机学院)
MMIR-TCM: Memory-Integrated Multimodal Inference and Retrieval for TCM Clinical Decision Support
Institute of Biopharmaceutics and Health Engineering, Tsinghua Shenzhen International Graduate School(清华深圳国际研究生院生物医药与健康工程研究院);Chinese Medicine Guangdong Laboratory(广东省中医药实验室);Beijing Normal University(北京师范大学);First Hospital of Hebei Medical University(河北医科大学第一医院);Lishui Hospital of Zhejiang University(浙江大学丽水医院);XiaoMing TCM Hospital(小明中医医院)
Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine
Harvard Medical School(哈佛医学院);Stanford University(斯坦福大学);Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University(哈佛大学约翰·A·保尔森工程与应用科学学院)
Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis
University Hospital Aachen(亚琛大学医院);RWTH Aachen University(亚琛工业大学);Ocumeda GmbH(Ocumeda 有限公司);RWTH Aachen(亚琛工业大学);Charité – Universitätsmedizin Berlin(柏林夏里特医学院);Freie Universität Berlin(柏林自由大学);Technical University Dresden(德累斯顿工业大学);University of Leeds(利兹大学);University Hospital Heidelberg(海德堡大学医院)
EHWGesture -- A dataset for multimodal understanding of clinical gestures
Department of Control and Computer Engineering, Politecnico di Torino(控制与计算机工程系,都灵理工大学);Fondazione Bruno Kessler(布鲁诺·凯斯勒基金会);CNR-IEIIT(意大利国家研究委员会-IEIIT)
Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications
Radboud University Medical Center(拉德堡德大学医学中心);Oncode Institute(Oncode研究所);Polytechnic University of Turin(都灵理工大学);University of Turin(都灵大学);Erasmus University Medical Centre(伊拉斯姆斯大学医学中心)
Multimodal Ordinal Modeling of Alzheimer's Disease Severity Using Structural MRI and Clinical Data
Department of Neuroradiology, LMU University Hospital, Ludwig Maximilian University of Munich(神经放射科,慕尼黑路德维希-马克西米利安大学医院,慕尼黑路德维希-马克西米利安大学);Department of Psychiatry and Psychotherapy, LMU University Hospital, Ludwig Maximilian University of Munich(精神病学与心理治疗系,慕尼黑路德维希-马克西米利安大学医院,慕尼黑路德维希-马克西米利安大学)
ClinicalFMamba: Advancing Clinical Assessment using Mamba-based Multimodal Neuroimaging Fusion
TD Bank Group, Toronto, Canada(TD银行集团,加拿大);Department of Computer Science, University of Toronto, Toronto, Canada(多伦多大学计算机科学系,加拿大);Department of Medical Imaging, University of Toronto, Toronto, Canada(多伦多大学医学影像系,加拿大)
Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer
Research Unit of Computer Systems and Bioinformatics, Department of Engineering(计算机系统与生物信息学研究单位,工程系);Research Unit of Radiation Oncology, Department of Medicine and Surgery(放射肿瘤学研究单位,医学与外科系);Research Unit of Anatomical Pathology, Department of Medicine and Surgery(病理学研究单位,医学与外科系)
Multimodal Deep Learning for Phyllodes Tumor Classification from Ultrasound and Clinical Data
Department of Electrical and Computer Engineering, University of Central Florida(电子与计算机工程系,中央佛罗里达大学);Massachusetts General Hospital, Department of Surgery, Section of Breast Surgery(麻省总医院,外科部,乳腺外科);Department of Medicine, University of Central Florida College of Medicine(医学系,中央佛罗里达大学医学院)