Healthcare data now span EHRs, medical imaging, genomics, and wearable sensors, but most diagnostic models still process these modalities in isolation. This limits their ability to capture early, cross-modal disease signatures. This paper introduces a multimodal foundation model built on a transformer architecture that integrates heterogeneous clinical data through modality-specific encoders and cross-modal attention. Each modality is mapped into a shared latent space and fused using multi-head attention with residual normalization. We implement the framework using a multimodal dataset that simulates early-stage disease patterns across EHR sequences, imaging patches, genomic profiles, and wearable signals, including missing-modality scenarios and label noise. The model is trained using supervised classification together with self-supervised reconstruction and contrastive alignment to improve robustness. Experimental evaluation demonstrates strong performance in early-detection settings, with stable classification metrics, reliable uncertainty estimates, and interpretable attention patterns. The approach moves toward a flexible, pretrain-and-fine-tune foundation model that supports precision diagnostics, handles incomplete inputs, and improves early disease detection across oncology, cardiology, and neurology applications.
Multimodal Learning on Graphs for Disease Relation Extraction
Beijing Institute of Technology(北京理工大学);University of Southern California(南加州大学);Tsinghua University(清华大学);Harvard T.H.Chan School of Public Health(哈佛陈曾熙公共卫生学院);Harvard Medical School(哈佛医学院);Broad Institute of MIT and Harvard(麻省理工学院和哈佛大学博德研究所);Harvard Data Science Initiative(哈佛数据科学研究院)
Flexible Multimodal Neuroimaging Fusion for Alzheimer's Disease Progression Prediction
Department of Computer Science and Engineering(计算机科学与工程系);Department of Biomedical Informatics(生物医学信息学系);Department of Neurology(神经病学系);Translational Data Analytics Institute(转化数据分析研究所)
Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis
Penn State University(宾夕法尼亚州立大学);Mississippi State University(密西西比州立大学);Northwest Missouri State University(西北密苏里州立大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of California, Berkeley(加利福尼亚大学伯克利分校);Colorado State University(科罗拉多州立大学)
Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Models
Hong Kong Baptist University(香港 Baptist 大学);Shanxi University(山西大学);Shanghai Institute for Advanced Study of Zhejiang University(浙江大学上海先进研究院);Hong Kong University of Science and Technology(香港科技大学);The University of Manchester(曼彻斯特大学)
CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models
Department of Computer Science and Engineering(计算机科学与工程系);Incheon National University(庆尚国立大学);Department of Artificial Intelligence(人工智能系);Inha University(Inha大学);Center for Brain-Machine Interface(脑机接口中心)
Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models
Technical University of Munich(慕尼黑工业大学);Hong Kong Baptist University(香港浸会大学);Aalto University(阿尔托大学);University of Oulu(奥卢大学);University of Bonn(波恩大学);EPFL(洛桑联邦理工学院)
A Sparse-Attention Deep Learning Model Integrating Heterogeneous Multimodal Features for Parkinson's Disease Severity Profiling
School of Computing, Mathematics, and Engineering, Charles Sturt University(计算、数学与工程学院,查尔斯·斯特劳特大学);School of Dentistry and Medical Sciences, Charles Sturt University(牙科学院与医学科学学院,查尔斯·斯特劳特大学);AI and Cyber Futures Centre (AICF)(人工智能与未来技术中心(AICF));Hawkins Clinic General Medical Practice(霍金斯诊所普通医疗实践)
Multimodal foundation models are better simulators of the human brain
Renmin University of China(中国人民大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);The University of Hong Kong(香港大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
多模态学习,特别是大规模的多模态预训练,在过去几年中发展迅速,并导致了人工智能(AI)的最大进步。尽管它的有效性,了解多模式预训练模型的潜在机制仍然是一个巨大的挑战。揭示这些模型的可解释性可能会使人工智能领域的新学习范式取得突破。为此,鉴于人类大脑的多模态性质,我们建议借助非侵入性脑成像技术(如功能磁共振成像(fMRI))来探索多模态学习模型的可解释性。具体地说,我们首先提出了一个新设计的多模态基础模型预训练的15万图像-文本对,它已显示出强大的多模态理解和泛化能力,在各种认知下游任务。此外,从神经编码的角度来看(基于我们的基础模型),我们发现多模态训练的视觉和语言编码器与单峰编码器相比更像大脑。特别是,我们确定了一些大脑区域,其中多模态训练的编码器表现出更好的神经编码性能。这与现有探索大脑多感觉整合的研究结果一致。因此,我们认为,多模态基础模型是更适合的工具,神经科学家研究的多模态信号处理机制在人脑中。我们的研究结果还证明了多模态基础模型作为理想的计算模拟器的潜力,以促进AI for Brain和Brain-for-AI研究。
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
InertialAI;Department of Electrical and Computer Engineering, Queen’s University(皇后大学电气与计算机工程系);Department of Mechanical and Materials Engineering, Queen’s University(皇后大学机械与材料工程系)
Large Language Multimodal Models for 5-Year Chronic Disease Cohort Prediction Using EHR Data
School of Systems and Enterprises, Stevens Institute of Technology(史蒂文斯理工学院系统与企业学院);National Yang Ming Chiao Tung University(国立阳明交通大学);Institute of Hospital and Health Care Administration, National Yang Ming Chiao Tung University(国立阳明交通大学医院暨健康保健管理研究所);School of Computing and Information Science, Florida International University(佛罗里达国际大学计算与信息科学学院);University of Michigan(密歇根大学);Far Eastern Memorial Hospital(远东纪念医院)
CHARM: A Multimodal Graph Foundation Model with Hierarchical Context Modeling for Zero-Shot Transfer
School of Computer Science and Technology, College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部计算机科学与技术学院);Data Science Program, Columbian College of Arts and Sciences, The George Washington University(乔治华盛顿大学文理学院哥伦比亚艺术与科学学院数据科学项目)
Captions Speak Louder than Images: Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data
Department of Computer Science and Engineering, The Ohio State University(计算机科学与工程系,俄亥俄州立大学);Translational Data Analytics Institute, The Ohio State University(转化数据分析研究所,俄亥俄州立大学);Department of Biomedical Informatics, The Ohio State University(生物医学信息学系,俄亥俄州立大学)
LeafNet: A Large-Scale Dataset and Comprehensive Benchmark for Foundational Vision-Language Understanding of Plant Diseases
School of Electrical Engineering, Korea University(韩国大学电气工程学院);Department of Integrative Biology, The University of Texas at Austin(德克萨斯大学奥斯汀分校整合生物学系);Department of Software Engineering, FPT University(FPT大学软件工程系)
Multimodal Audio-based Disease Prediction with Transformer-based Hierarchical Fusion Network
Purdue University(普渡大学);Beijing University of Chemical Technology(北京化工大学);New York University(纽约大学);University of Cincinnati(辛辛那提大学);Indiana University(印第安纳大学)
Multimodal Ordinal Modeling of Alzheimer's Disease Severity Using Structural MRI and Clinical Data
Department of Neuroradiology, LMU University Hospital, Ludwig Maximilian University of Munich(神经放射科,慕尼黑路德维希-马克西米利安大学医院,慕尼黑路德维希-马克西米利安大学);Department of Psychiatry and Psychotherapy, LMU University Hospital, Ludwig Maximilian University of Munich(精神病学与心理治疗系,慕尼黑路德维希-马克西米利安大学医院,慕尼黑路德维希-马克西米利安大学)
Towards Multimodal Representation Learning in Paediatric Kidney Disease
Institute of Health Informatics(健康信息学研究所);University College London(伦敦大学学院);Data Research, Innovation and Virtual Environments Unit(数据研究、创新与虚拟环境单位);Great Ormond Street Hospital(格雷特奥蒙德医院);Department of Computer Science(计算机科学系)
Graph Attention Based Prioritization of Disease Responsible Genes from Multimodal Alzheimer's Network
Network Reconstruction & Analysis Lab and Department of Computer Applications, Sikkim University(网络重建与分析实验室和计算机应用系,西西米大学);Network Reconstruction & Analysis Lab, Department of Computer Applications, Sikkim University(网络重建与分析实验室,计算机应用系,西西米大学);Department of Computer Science & Engineering, Tezpur University(计算机科学与工程系,泰朱大学)
A Generative Imputation Method for Multimodal Alzheimer's Disease Diagnosis
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA(电气与计算机工程学院,佐治亚理工学院,亚特兰大,GA,USA);Courtesy Faculty Appointment, College of Pharmacy, University of Florida, Gainesville, FL, USA(佛罗里达大学药学院荣誉教职)
BrainMAP: Multimodal Graph Learning For Efficient Brain Disease Localization
School of Computing Technologies, RMIT University(计算技术学院,拉筹纳斯大学);Institute of Innovation, Science and Sustainability, Federation University Australia(创新、科学与可持续性研究所,联邦大学澳大利亚)
A Self-guided Multimodal Approach to Enhancing Graph Representation Learning for Alzheimer's Diseases
George Mason University(乔治梅森大学);GE Healthcare(GE医疗);University of Pittsburgh(匹兹堡大学);University of Maryland(马里兰大学);Southern University of Science and Technology(南方科技大学);William and Mary(威廉玛丽学院)
OPTIMUS: Predicting Multivariate Outcomes in Alzheimer's Disease Using Multi-modal Data amidst Missing Values
University Lausanne Hospital (CHUV)(洛桑大学医院);University of Lausanne (UNIL)(洛桑大学);Zhejiang University(浙江大学);Zhejiang University School of Medicine(浙江大学医学院)
Self-Supervised Multimodal Domino: in Search of Biomarkers for Alzheimer's Disease
Georgia Institute of Technology(佐治亚理工学院);Georgia State University(佐治亚州立大学);Emory University(埃默里大学);Center for Translational Research in Neuroimaging and Data Science(神经影像学与数据科学转化研究中心);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Delft University of Technology(代尔夫特理工大学)
Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine
Harvard Medical School(哈佛医学院);Stanford University(斯坦福大学);Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University(哈佛大学约翰·A·保尔森工程与应用科学学院)