arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2602.12499cs.LG
收藏

Mamba训练动态的理论分析:在状态空间模型中筛选相关特征以实现泛化

A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Models

New Jersey Institute of Technology(新泽西理工学院);University of Pennsylvania(宾夕法尼亚大学);University of Louisiana at Lafayette(路易斯安那州立大学拉法叶分校)

Mugunthan Shandirasegaran, Hongkang Li, Songyang Zhang, Meng Wang, Shuai Zhang

AI总结:

本文分析了Mamba模型的训练动态,证明其通过选择性递归实现特征筛选,提升泛化能力,为Transformer解释提供了理论依据。

摘要:

最近Mamba和其他选择性状态空间模型(SSMs)的实证成功重新激发了对非注意力架构在序列建模中的兴趣,但其理论基础仍待深入研究。我们对一个简化但具有代表性的Mamba块进行了初步分析:一个单层、单头的选择性SSM,具有输入依赖的门控,随后通过梯度下降(GD)训练的两层MLP。我们的研究采用了一种结构化的数据模型,其中的token既包含与类别相关的模式也包含与类别无关的模式,并在token层面存在噪声。我们考察了两种典型情形:多数投票和局部结构化的数据序列。我们证明该模型通过建立非渐近的样本复杂度和收敛速率界来实现保证的泛化,这些界随着有效信号的增加和噪声的减少而改善。此外,我们显示门控向量与类别相关特征对齐,同时忽略无关特征,从而正式化了一种类似于注意力的特征选择角色,但通过选择性递归实现。在合成数据上的数值实验验证了我们的理论结果。总体而言,我们的结果提供了关于何时以及为何Mamba式选择性SSM高效学习的原理性见解,为Transformer中心的解释提供了理论反驳。

英文摘要:

The recent empirical success of Mamba and other selective state space models (SSMs) has renewed interest in non-attention architectures for sequence modeling, yet their theoretical foundations remain underexplored. We present a first-step analysis of generalization and learning dynamics for a simplified but representative Mamba block: a single-layer, single-head selective SSM with input-dependent gating, followed by a two-layer MLP trained via gradient descent (GD). Our study adopts a structured data model with tokens that include both class-relevant and class-irrelevant patterns under token-level noise and examines two canonical regimes: majority-voting and locality-structured data sequences. We prove that the model achieves guaranteed generalization by establishing non-asymptotic sample complexity and convergence rate bounds, which improve as the effective signal increases and the noise decreases. Furthermore, we show that the gating vector aligns with class-relevant features while ignoring irrelevant ones, thereby formalizing a feature-selection role similar to attention but realized through selective recurrence. Numerical experiments on synthetic data justify our theoretical results. Overall, our results provide principled insight into when and why Mamba-style selective SSMs learn efficiently, offering a theoretical counterpoint to Transformer-centric explanations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑