解析多模态上下文学习：现代Transformer中的模态不对称性与电路动力学
Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers
AI总结：
通过可控实验，研究现代Transformer中多模态上下文学习的基本机制，发现模态间学习不对称性，并揭示其背后的归纳式电路机制。
摘要：
基于Transformer的多模态大语言模型通常展现出上下文学习（ICL）能力。受此现象启发，我们提出疑问：Transformer如何从上下文示例中跨模态关联信息？我们通过在合成分类任务上训练的小型Transformer进行可控实验来研究这一问题，从而能够精确操控数据统计和模型架构。我们首先重新审视现代Transformer中单模态ICL的核心原理。虽然多个先前发现得以复现，但我们发现旋转位置编码（RoPE）提高了ICL的数据复杂度阈值。扩展到多模态设置揭示了一个基本的学习不对称性：当在来自主要模态的高多样性数据上预训练时，次要模态中令人惊讶的低数据复杂度就足以使多模态ICL出现。机制分析表明，两种设置都依赖于一种归纳式机制，该机制从匹配的上下文示例中复制标签；多模态训练则跨模态细化和扩展这些电路。我们的发现为理解现代Transformer中的多模态ICL提供了机制基础，并为未来研究引入了一个可控的测试平台。代码可在 https://github.com/YiranHuangIrene/multimodal-icl 获取。
英文摘要：
Transformer-based multimodal large language models often exhibit in-context learning (ICL) abilities. Motivated by this phenomenon, we ask: how do transformers learn to associate information across modalities from in-context examples? We investigate this question through controlled experiments on small transformers trained on synthetic classification tasks, enabling precise manipulation of data statistics and model architecture. We begin by revisiting core principles of unimodal ICL in modern transformers. While several prior findings replicate, we find that Rotary Position Embeddings (RoPE) increases the data complexity threshold for ICL. Extending to the multimodal setting reveals a fundamental learning asymmetry: when pretrained on high-diversity data from a primary modality, surprisingly low data complexity in the secondary modality suffices for multimodal ICL to emerge. Mechanistic analysis shows that both settings rely on an induction-style mechanism that copies labels from matching in-context exemplars; multimodal training refines and extends these circuits across modalities. Our findings provide a mechanistic foundation for understanding multimodal ICL in modern transformers and introduce a controlled testbed for future investigation. Code is available at: https://github.com/YiranHuangIrene/multimodal-icl
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
