arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.19047cs.LGmath.DS
收藏

深度选择性状态空间模型的理论基础

Theoretical Foundations of Deep Selective State-Space Models

Imperial College London(伦敦帝国学院);MPI for Intelligent Systems(马克斯·普朗克智能系统研究所);Tübingen AI Center(图宾根人工智能中心);ELLIS Institute Tübingen(ELLIS图宾根研究所);University of Oxford(牛津大学)

Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, Terry Lyons

AI总结:

本文借助粗糙路径理论证明,带输入控制选择性机制的随机线性递归其隐藏状态是输入 signature 的低维投影,从而为 Mamba 等选择性 SSM 的表达能力提供理论解释。

摘要:

结构化状态空间模型(SSM),例如源自 Gu 等人开创性工作的 S4,正作为建模序列数据的有效方法而受到欢迎。与基于注意力的 transformer 相比,深度 SSM 在多种不同领域中表现出卓越性能,同时训练和推理成本更低。近期进展表明,如果驱动 SSM 的线性递归允许输入与隐藏状态之间存在乘法交互(例如 GateLoop、Mamba、GLA),那么所得架构在十亿参数规模上,在准确性和效率两方面都能超过基于注意力训练的文本基础模型。本文中,我们利用粗糙路径理论(Rough Path Theory)的工具为这一近期发现提供理论基础:我们证明,当随机线性递归配备简单的输入控制转移(选择性机制)时,隐藏状态可被证明是一个强大数学对象——输入的 signature——的低维投影,该对象捕捉不同时间尺度上 token 之间的非线性交互。我们的理论不仅解释了 Mamba 等现代选择性状态空间模型成功的原因,也为理解未来 SSM 变体的表达能力提供了坚实框架。

英文摘要:

Structured state-space models (SSMs) such as S4, stemming from the seminal work of Gu et al., are gaining popularity as effective approaches for modeling sequential data. Deep SSMs demonstrate outstanding performance across a diverse set of domains, at a reduced training and inference cost compared to attention-based transformers. Recent developments show that if the linear recurrence powering SSMs allows for multiplicative interactions between inputs and hidden states (e.g. GateLoop, Mamba, GLA), then the resulting architecture can surpass in both in accuracy and efficiency attention-powered foundation models trained on text, at scales of billion parameters. In this paper, we give theoretical grounding to this recent finding using tools from Rough Path Theory: we show that when random linear recurrences are equipped with simple input-controlled transitions (selectivity mechanism), then the hidden state is provably a low-dimensional projection of a powerful mathematical object called the signature of the input -- capturing non-linear interactions between tokens at distinct timescales. Our theory not only motivates the success of modern selective state-space models such as Mamba but also provides a solid framework to understand the expressive power of future SSM variants.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑