We use (multi)modal deep neural networks (DNNs) to probe for sites of multimodal integration in the human brain by predicting stereoencephalography (SEEG) recordings taken while human subjects watched movies. We operationalize sites of multimodal integration as regions where a multimodal vision-language model predicts recordings better than unimodal language, unimodal vision, or linearly-integrated language-vision models. Our target DNN models span different architectures (e.g., convolutional networks and transformers) and multimodal training techniques (e.g., cross-attention and contrastive learning). As a key enabling step, we first demonstrate that trained vision and language models systematically outperform their randomly initialized counterparts in their ability to predict SEEG signals. We then compare unimodal and multimodal models against one another. Because our target DNN models often have different architectures, number of parameters, and training sets (possibly obscuring those differences attributable to integration), we carry out a controlled comparison of two models (SLIP and SimCLR), which keep all of these attributes the same aside from input modality. Using this approach, we identify a sizable number of neural sites (on average 141 out of 1090 total sites or 12.94%) and brain regions where multimodal integration seems to occur. Additionally, we find that among the variants of multimodal training techniques we assess, CLIP-style training is the best suited for downstream prediction of the neural activity in these sites.
Supervised Deep Multimodal Matrix Factorization for Interpretable Brain Network Analysis
Dept. of Mathematics & Operational Research University of Mons（数学与运筹学系蒙斯大学）；Dept. of Computer Science & Engineering Lehigh University（计算机科学与工程系莱斯大学）；Dept. of Industrial & Systems Engineering Lehigh University（工业与系统工程系莱斯大学）
Explainable concept mappings of MRI: Revealing the mechanisms underlying deep learning-based brain disease classification
Medical University of Graz（格拉茨医科大学）；Monash University（莫纳什大学）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Technische Universität Berlin（柏林工业大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（BIFOLD——柏林学习与数据基础研究所）
CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models
Department of Computer Science and Engineering（计算机科学与工程系）；Incheon National University（庆尚国立大学）；Department of Artificial Intelligence（人工智能系）；Inha University（Inha大学）；Center for Brain-Machine Interface（脑机接口中心）
Self-supervised multimodal neuroimaging yields predictive representations for a spectrum of Alzheimer's phenotypes
Tri-Institutional Center for Translational Research in Neuroimaging and Data Science（三机构神经影像与数据科学转化研究中心）；Georgia State（佐治亚州立大学）；Georgia Tech（佐治亚理工学院）；Emory（埃默里大学）；Borealis AI（Borealis人工智能公司）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Apple Machine Learning Research（苹果机器学习研究院）；Université de Montréal（蒙特利尔大学）
ClusMFL: A Cluster-Enhanced Framework for Modality-Incomplete Multimodal Federated Learning in Brain Imaging Analysis
School of Science and Engineering, the Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；Department of Computer Science and Engineering, Lehigh University（利哈伊大学计算机科学与工程系）；Department of Computer Science, Emory University（埃默里大学计算机科学系）
Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
Technische Universität Berlin（柏林技术大学）；IIIT Hyderabad（海得拉巴国家理工学院）；Univ of Maryland（马里兰大学）；Rice Univ（Rice 大学）；Univ of Wisconsin - Madison（威斯康星大学麦迪逊分校）；Spector Inc（Spector 公司）；Microsoft（微软公司）
BrainMAP: Multimodal Graph Learning For Efficient Brain Disease Localization
School of Computing Technologies, RMIT University（计算技术学院，拉筹纳斯大学）；Institute of Innovation, Science and Sustainability, Federation University Australia（创新、科学与可持续性研究所，联邦大学澳大利亚）
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；School of Electrical and Information Engineering, Zhengzhou University（郑州大学电气工程学院）
BrainSymphony: A parameter-efficient multimodal foundation model for brain dynamics with limited data
Turner Institute for Brain and Mental Health（大脑与心理健康Turner研究所）；School of Psychological Sciences, Monash University（墨尔本大学心理学科学学院）；Cortical Labs（皮层实验室）；CIFAR Azrieli Global Scholars Program（CIFAR阿兹里埃利全球学者计划）
Brain-MGF: Multimodal Graph Fusion Network for EEG-fMRI Brain Connectivity Analysis Under Psilocybin
School of Information Technology, Monash University Malaysia（信息技术学院，墨尔本大学马来西亚分校）；Department of Data Science and AI, Faculty of Information Technology, Monash University（数据科学与人工智能系，信息科技学院，墨尔本大学）；Turner Institute for Brain and Mental Health, School of Psychological Sciences, Monash University（脑与心理健康研究所，心理学科学学院，墨尔本大学）
Revealing Multimodal Causality with Large Language Models
University of Technology Sydney（技术大学悉尼大学）；Tongji University（同济大学）；University of Melbourne（墨尔本大学）；University of Sydney（悉尼大学）；Macquarie University（麦考瑞大学）
Joint Embedding of Structural and Functional Brain Networks with Graph Neural Networks for Mental Illness Diagnosis
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Emory University（埃默里大学）；Lehigh University（理海大学）
Multimodal Recurrent Ensembles for Predicting Brain Responses to Naturalistic Movies (Algonauts 2025)
Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；TU Dresden（德累斯顿技术大学）；School for Embedded and Composite AI (SECAI)（嵌入式与复合人工智能学院）；Center for Scalable Data Analytics & AI (ScaDS.AI)（可扩展数据与人工智能中心）
Interpretable multimodal fusion networks reveal mechanisms of brain cognition
Tulane University（杜兰大学）；Central South University（中南大学）；Mind Research Network（心智研究网络）；University of Nebraska Medical Center（内布拉斯加大学医学中心）；Georgia State University（佐治亚州立大学）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）
Fourier Disentangled Multimodal Prior Knowledge Fusion for Red Nucleus Segmentation in Brain MRI
Sorbonne Université（索邦大学）；Institut du Cerveau - Paris Brain Institute - ICM（巴黎大脑研究所（ICM））；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Inserm（法国国家健康与医学研究院）；AP-HP（巴黎公共援助医院集团）；Hôpital de la Pitié Salpêtrière（皮蒂耶-萨尔佩特里埃医院）
A Few-shot Learning Graph Multi-Trajectory Evolution Network for Forecasting Multimodal Baby Connectivity Development from a Baseline Timepoint
Istanbul Technical University（伊斯坦布尔技术大学）；University of Sousse（苏塞大学）；National School for Computer Science (ENSI)（国家计算机科学学校）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；ShanghaiTech University（上海科技大学）；United Imaging Intelligence Co., Ltd.（联影智能有限公司）
