曼巴如何在有异常值的情况下进行上下文学习并实现可证明的泛化？
How Can Mamba Learn In Context with Outliers and Generalize Provably?
AI总结：
研究曼巴模型在有异常值时的上下文学习及泛化能力，通过分析其线性注意力层和非线性门控层，发现能选信息示例、抑异常值影响，虽收敛迭代多，但异常值超阈值时仍能准确预测，理论获实证支持。
摘要：
曼巴模型因其相对于基于Transformer的模型的计算优势而备受关注，在广泛的语言任务中表现相当。与Transformer一样，曼巴具有上下文学习（ICL）能力。尽管其实验成功，但对曼巴的理论理解仍然有限。本文首次对单层曼巴模型的训练动态及其在未见二元分类任务上的ICL泛化进行理论分析，即使提示包含加性异常值。分析表明，曼巴利用线性注意力层选择信息性上下文示例，并用非线性门控层抑制异常值的影响。通过与相同设置下的线性Transformer分析比较，发现曼巴虽可能需要更多训练迭代来收敛，但在异常值比例超过线性Transformer可容忍阈值时仍能保持准确预测，这些理论发现得到了实证实验的支持。
英文摘要：
The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks. Like Transformers, Mamba exhibits in-context learning (ICL) capabilities, i.e., making predictions for new tasks based on a prompt containing input-label pairs and a query, without requiring fine-tuning. Despite its empirical success, the theoretical understanding of Mamba remains limited, largely due to the nonlinearity introduced by its gating mechanism. To the best of our knowledge, this paper presents the first theoretical analysis of the training dynamics of a one-layer Mamba model, which consists of a linear attention component followed by a nonlinear gating layer, and its ICL generalization on unseen binary classification tasks, even when the prompt includes additive outliers. Our analysis shows that Mamba leverages the linear attention layer to select informative context examples and uses the nonlinear gating layer to suppress the influence of outliers. By establishing and comparing to the analysis of linear Transformers under the same setting, we show that although Mamba may require more training iterations to converge, it maintains accurate predictions even when the proportion of outliers exceeds the threshold that a linear Transformer can tolerate. These theoretical findings are supported by empirical experiments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
