Recent progress has rapidly advanced our understanding of the mechanisms underlying in-context learning in modern attention-based neural networks. However, existing results focus exclusively on unimodal data; in contrast, the theoretical underpinnings of in-context learning for multi-modal data remain poorly understood. We introduce a mathematically tractable framework for studying multi-modal learning and explore when transformer-like architectures can recover Bayes-optimal performance in-context. To model multi-modal problems, we assume the observed data arises from a latent factor model. Our first result comprises a negative take on expressibility: we prove that single-layer self-attention models fail to recover the Bayes-optimal predictor uniformly over the task distribution. To address this limitation, we introduce a novel, linearized cross-attention mechanism, which we study in the regime where both the number of cross-attention layers and the context length are large. We show that this cross-attention mechanism is provably Bayes optimal when optimized using gradient flow. Our results underscore the benefits of depth for in-context learning and establish the provable utility of cross-attention for multi-modal distributions.
Center for Foundation Models and Generative AI & Department of Computer Science, Northwestern University, USA(基础模型与生成人工智能中心及计算机科学系,西北大学,美国);School of Mathematical Sciences, Fudan University(复旦大学数学科学学院);University of California, Berkeley, USA(加州大学伯克利分校,美国);Center for Foundation Models and Generative AI & Department of Computer Science & Department of Statistics and Data Science, Northwestern University, USA(基础模型与生成人工智能中心及计算机科学系与统计与数据科学系,西北大学,美国)
Transformers Provably Learn Directed Acyclic Graphs via Kernel-Guided Mutual Information
Department of Mathematics, National University of Singapore, Singapore(新加坡国立大学数学系);Department of Statistics and Data Science, University of Pennsylvania, USA(宾夕法尼亚大学统计与数据科学系);Department of Electrical and Computer Engineering, The Ohio State University, USA(俄亥俄州立大学电子与计算机工程系)
Multi-Modal Opinion Integration for Financial Sentiment Analysis using Cross-Modal Attention
College of Computing Georgia Institute of Technology Atlanta, USA(计算学院 佐治亚理工学院 美国亚特兰大);College of Engineering University of Pennsylvania Philadelphia, USA(工程学院 宾夕法尼亚大学 美国费城)
MolFM-Lite: Multi-Modal Molecular Property Prediction with Conformer Ensemble Attention and Cross-Modal Fusion
Department of Computer Science and Engineering, University at Buffalo(布法罗大学计算机科学与工程系);Khoury College of Computer Sciences, Northeastern University(东北大学科赫里学院);Department of Computer Science, Muffakham Jah College of Engineering and Technology(穆法卡姆·贾赫工程与技术学院计算机科学系);Department of Computer Science and Artificial Intelligence, Muffakham Jah College of Engineering and Technology(穆法卡姆·贾赫工程与技术学院计算机科学与人工智能系)
Multimodal Inverse Attention Network with Intrinsic Discriminant Feature Exploitation for Fake News Detection
School of Information Science and Engineering, Shandong Normal University(山东师范大学信息科学与工程学院);Faculty of Engineering & Information Technology, University of Technology Sydney(悉尼科技大学工程与信息技术学院)