arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.01959cs.LG
收藏

对状态跟踪任务中对角复值DCD SSMs表达能力的限制

The Expressive Limits of Diagonal SSMs for State-Tracking

McGill University(麦吉尔大学);Mila – Quebec AI Institute(魁北克AI研究所);Polytechnique Montréal(蒙特利尔大学)

Mehran Shakerinava, Behnoush Khavari, Siamak Ravanbakhsh, Sarath Chandar

AI总结:

本文研究了对角复值DCD SSMs在状态跟踪任务中的表达能力,证明了其在有限精度下无法表达非阿贝尔群的状态跟踪,并揭示了表达能力与可学习性之间的差距。

摘要:

状态空间模型(SSMs)最近已被证明在多种长序列建模任务中表现出强大的经验性能,同时保持高效和高度并行化。然而,对其表达能力的理论理解仍然有限。在本工作中,我们研究了输入依赖复值对角(DCD)SSMs在顺序状态跟踪任务中的表达性。我们证明单层DCD SSMs无法在有限精度下表达任何非阿贝尔群的状态跟踪。更一般地说,我们证明k层DCD SSMs能够表达一个群的状态跟踪当且仅当该群有一个长度为k的子正规列,且每个因子都是阿贝尔群。也就是说,我们确定了k层DCD SSMs在可解群中的精确表达能力范围。实证上,我们发现多层模型经常无法学习非阿贝尔群的状态跟踪,突显了表达能力和可学习性之间的差距。

英文摘要:

State-Space Models (SSMs) have recently been shown to achieve strong empirical performance on a variety of long-range sequence modeling tasks while remaining efficient and highly-parallelizable. However, the theoretical understanding of their expressive power remains limited. In this work, we study the expressivity of input-Dependent Complex-valued Diagonal (DCD) SSMs on sequential state-tracking tasks. We show that single-layer DCD SSMs cannot express state-tracking of any non-Abelian group at finite precision. More generally, we show that $k$-layer DCD SSMs can express state-tracking of a group if and only if that group has a subnormal series of length $k$, with Abelian factors. That is, we identify the precise expressivity range of $k$-layer DCD SSMs within the solvable groups. Empirically, we find that multi-layer models often fail to learn state-tracking for non-Abelian groups, highlighting a gap between expressivity and learnability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑