State-Space Models (SSMs) have recently been shown to achieve strong empirical performance on a variety of long-range sequence modeling tasks while remaining efficient and highly-parallelizable. However, the theoretical understanding of their expressive power remains limited. In this work, we study the expressivity of input-Dependent Complex-valued Diagonal (DCD) SSMs on sequential state-tracking tasks. We show that single-layer DCD SSMs cannot express state-tracking of any non-Abelian group at finite precision. More generally, we show that $k$-layer DCD SSMs can express state-tracking of a group if and only if that group has a subnormal series of length $k$, with Abelian factors. That is, we identify the precise expressivity range of $k$-layer DCD SSMs within the solvable groups. Empirically, we find that multi-layer models often fail to learn state-tracking for non-Abelian groups, highlighting a gap between expressivity and learnability.
On the Expressive Power and Limitations of Multi-Layer SSMs
Robotics and Perception Group(机器人与感知组);University of Zurich(苏黎世大学);Shenzhen International Center For Industrial And Applied Mathematics(深圳国际工业与应用数学中心);Shenzhen Research Institute of Big Data(深圳大数据研究 institute);Tengen Intelligence Institute(腾格智能研究所);CRRC Zhuzhou Institute(CRRC 长沙研究所)
Parity Requires Unified Input Dependence and Negative Eigenvalues in SSMs
Chandar Research Lab(昌达尔研究实验室);Mila - Quebec AI Institute(蒙特利尔AI研究所);Queens University(皇后大学);CIFAR AI Chair(CIFAR人工智能主席);McGill University(麦吉尔大学);Royal Military College of Canada(加拿大皇家军事学院)
On the Parameterization and Initialization of Diagonal State Space Models
Stanford University(斯坦福大学);IBM Research(IBM研究院)
状态空间模型(SSM)最近被证明是非常有效的深度学习层,是RNN,CNN或Transformers等序列模型的有前途的替代品。第一个显示这种潜力的版本是S4模型,它通过使用称为HiPPO矩阵的规定状态矩阵,对涉及远程依赖关系的任务特别有效。虽然这有一个可解释的数学机制来建模长依赖关系,但它引入了一个难以实现的自定义表示和算法。另一方面,最近的一个称为DSS的S4变体表明,当使用基于近似S4矩阵的特定初始化时,将状态矩阵限制为全对角时,仍然可以保持原始模型的性能。这项工作旨在系统地了解如何参数化和初始化这样的对角状态空间模型。虽然它遵循经典的结果,几乎所有的SSM有一个等价的对角形式,我们表明,初始化是至关重要的性能。我们解释了为什么DSS在数学上工作,通过显示S4的矩阵的对角限制令人惊讶地恢复了相同的内核在无限状态维度的限制。我们还系统地描述了各种设计选择的参数化和计算对角SSM,并进行了控制的实证研究消融这些选择的影响。我们的最终模型S4D是S4的简单对角版本,其内核计算仅需2行代码,几乎在所有设置中都可以执行S4,图像,音频和医疗时间序列域的最新结果,在Long Range Arena基准测试中平均为85%。
Fixed-Point RNNs: Interpolating from Diagonal to Dense
ELLIS Institute Tuebingen(图宾根ELLIS研究所);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);Department of Mathematics, Imperial College London(伦敦帝国学院数学系)
DSSMs: State Space Models with Explicit Memory via Delay Differential Equations
College of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院);Department of Mathematics, National University of Singapore(新加坡国立大学数学系);School of Mathematical Sciences, Zhejiang University(浙江大学数学科学学院)
Robustifying State-space Models for Long Sequences via Approximate Diagonalization
Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);International Computer Science Institute(国际计算机科学研究所);Cornell University(康奈尔大学);University of California at Berkeley(加州大学伯克利分校)
Universal Time Series Generation with Neural Controlled Differential Equations
Heidelberg Institute for Theoretical Studies(海德堡理论研究所);IAR, Karlsruhe Institute of Technology(卡尔斯鲁厄技术大学IAR部门);Max Planck Institute for Polymer Research(马克斯·普朗克聚合物研究所);IWR, Heidelberg University(海德堡大学IWR部门);Dept. of Computer Science, University of Toronto(多伦多大学计算机科学系);Mathematical Institute, University of Oxford(牛津大学数学研究所);Vector Institute, Toronto, Canada(多伦多向量研究所)
Center for Foundation Models and Generative AI, Northwestern University(基础模型与生成式AI中心,西北大学);Department of Computer Science, Northwestern University(计算机科学系,西北大学);Ensemble AI;University of California, San Diego(加州大学圣地亚哥分校);Department of Statistics and Data Science, Northwestern University(统计与数据科学系,西北大学)
Deep Direct Discriminative Decoders for High-dimensional Time-series Data Analysis
Krembil Brain Institute, University Health Network(大学健康网络克雷姆比尔脑研究所);KITE Research Institute, Rehabilitation Institute, University Health Network(大学健康网络康复研究所KITE研究院);University of Toronto(多伦多大学);Boston University(波士顿大学);Worcester Polytechnic Institute (WPI)(伍斯特理工学院(WPI))
Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models
Intelligent Systems and Informatics Laboratory, Graduate School of Information Science and Technology, The University of Tokyo(智能系统与信息学实验室,信息科学与技术研究生院,东京大学)
Block-Biased Mamba for Long-Range Sequence Processing
Center for Applied Mathematics(应用数学中心);Cornell University(康奈尔大学);Lawrence Berkeley National Laboratory(伯克利国家实验室);International Computer Science Institute(国际计算机科学研究所)
Theoretical Foundations of Deep Selective State-Space Models
Imperial College London(伦敦帝国学院);MPI for Intelligent Systems(马克斯·普朗克智能系统研究所);Tübingen AI Center(图宾根人工智能中心);ELLIS Institute Tübingen(ELLIS图宾根研究所);University of Oxford(牛津大学)
The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity
The University of Chicago(芝加哥大学);University of New South Wales(新南威尔士大学);The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Simons Institute for the Theory of Computing at UC Berkeley(加州大学伯克利分校西蒙斯理论计算研究所)
Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity
Apple(苹果公司);Work done while at Apple(在苹果公司的工作);Mila Research Institute, Montreal, Canada(蒙特利尔加拿大米尔研究所);Flatiron Institute, New York, USA(纽约美国Flatiron研究所)
Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues
ELLIS Institute Tübingen(ELLIS研究所蒂宾根分部);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Tübingen AI Center(蒂宾根人工智能中心);Google Deepmind(谷歌DeepMind);EPFL(洛桑联邦理工学院)
High-dimensional Limit of SGD for Diagonal Linear Networks
University of Washington(华盛顿大学);McGill University(麦吉尔大学);Google DeepMind(谷歌DeepMind);University of Washington & Amazon Inc.(华盛顿大学与亚马逊公司);University of California, San Diego(加州大学圣地亚哥分校)