arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.14067cs.LGcs.AIcs.NE
收藏

结构化状态空间模型中复参数化的可证明优势

Provable Benefits of Complex Parameterizations for Structured State Space Models

Tel Aviv University(特拉维夫大学);Google(谷歌)

Yuval Ran-Milo, Eden Lumbroso, Edo Cohen-Karlik, Raja Giryes, Amir Globerson, Nadav Cohen

AI总结:

该研究针对结构化状态空间模型复参数化的理论优势难题,证明了复值对角SSM在表达能力和参数规模上显著优于实值模型,实验验证了理论并指出可扩展至选择性架构特征。

摘要:

结构化状态空间模型(SSMs)是S4、Mamba等知名神经网络背后的核心引擎,是遵循特定结构(最显著的是对角结构)的线性动力系统。与参数化为实数的典型神经网络模块不同,SSMs通常使用复参数化。从理论上解释复参数化对SSMs的益处是一个悬而未决的问题。本文通过建立实值对角SSM与复值对角SSM之间的形式化差距,为解决该问题迈出了一步。首先,我们证明:虽然中等维度的复值SSM就足以表达实值SSM的所有映射,但实值SSM需要高得多的维度才能表达复值SSM的映射。其次,我们证明:即使实值SSM的维度高到足以表达给定映射,通常这样做也需要实值SSM的参数取指数级大的值,这在实践中是无法学习的。相比之下,复值SSM可以用适中的参数值表达任意给定映射。实验验证了我们的理论,并暗示了该理论的一种潜在扩展,可用于解释选择性(selectivity)——一种实现了当前最优性能的新型架构特征。

英文摘要:

Structured state space models (SSMs), the core engine behind prominent neural networks such as S4 and Mamba, are linear dynamical systems adhering to a specified structure, most notably diagonal. In contrast to typical neural network modules, whose parameterizations are real, SSMs often use complex parameterizations. Theoretically explaining the benefits of complex parameterizations for SSMs is an open problem. The current paper takes a step towards its resolution, by establishing formal gaps between real and complex diagonal SSMs. Firstly, we prove that while a moderate dimension suffices in order for a complex SSM to express all mappings of a real SSM, a much higher dimension is needed for a real SSM to express mappings of a complex SSM. Secondly, we prove that even if the dimension of a real SSM is high enough to express a given mapping, typically, doing so requires the parameters of the real SSM to hold exponentially large values, which cannot be learned in practice. In contrast, a complex SSM can express any given mapping with moderate parameter values. Experiments corroborate our theory, and suggest a potential extension of the theory that accounts for selectivity, a new architectural feature yielding state of the art performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑