arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04872stat.MLcs.AIcs.LG
收藏

多层交叉注意力是多模态上下文学习中可证明最优的

Multi-layer Cross-attention is Provably Optimal for Multi-modal In-context Learning

Harvard University(哈佛大学)

Nicholas Barnfield, Subhabrata Sen, Pragya Sur

AI总结:

本文研究了多模态上下文学习中多层交叉注意力机制的理论最优性,证明了在多模态数据下,交叉注意力机制在梯度流优化下可达到贝叶斯最优,同时指出单层线性自注意力无法在任务分布下统一恢复贝叶斯最优预测。

摘要:

近期进展迅速推动了我们对现代基于注意力的神经网络中上下文学习机制的理解。然而,现有结果仅专注于单模态数据;相比之下,多模态数据的上下文学习的理论基础仍不清晰。我们引入了一个数学上可处理的框架来研究多模态学习,并探讨了在何种情况下Transformer-like架构可以在上下文中恢复贝叶斯最优性能。为了建模多模态问题,我们假设观测数据来自一个潜在因子模型。我们的第一个结果是对表达性的否定:我们证明单层线性自注意力无法在任务分布下统一恢复贝叶斯最优预测。为了解决这一限制,我们引入了一种新的线性化交叉注意力机制,并在交叉注意力层和上下文长度都较大的情况下进行研究。我们证明,当使用梯度流优化时,这种交叉注意力机制可证明是贝叶斯最优的。我们的结果强调了深度对上下文学习的好处,并确立了交叉注意力在多模态分布中的可证明效用。

英文摘要:

Recent progress has rapidly advanced our understanding of the mechanisms underlying in-context learning in modern attention-based neural networks. However, existing results focus exclusively on unimodal data; in contrast, the theoretical underpinnings of in-context learning for multi-modal data remain poorly understood. We introduce a mathematically tractable framework for studying multi-modal learning and explore when transformer-like architectures can recover Bayes-optimal performance in-context. To model multi-modal problems, we assume the observed data arises from a latent factor model. Our first result comprises a negative take on expressibility: we prove that single-layer self-attention models fail to recover the Bayes-optimal predictor uniformly over the task distribution. To address this limitation, we introduce a novel, linearized cross-attention mechanism, which we study in the regime where both the number of cross-attention layers and the context length are large. We show that this cross-attention mechanism is provably Bayes optimal when optimized using gradient flow. Our results underscore the benefits of depth for in-context learning and establish the provable utility of cross-attention for multi-modal distributions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑