On the Expressive Power and Limitations of Multi-Layer SSMs
Robotics and Perception Group(机器人与感知组);University of Zurich(苏黎世大学);Shenzhen International Center For Industrial And Applied Mathematics(深圳国际工业与应用数学中心);Shenzhen Research Institute of Big Data(深圳大数据研究 institute);Tengen Intelligence Institute(腾格智能研究所);CRRC Zhuzhou Institute(CRRC 长沙研究所)
Nikola Zubić, Qian Li, Yuyi Wang, Davide Scaramuzza
We study how depth, finite precision, state dimension, and chain-of-thought (CoT) affect the expressive power of multi-layer state-space models (SSMs). For the explicit-table $K$-function-composition problem, a canonical benchmark for sequential information propagation, we prove that any $L$-layer SSM solving $(L+3)$-function composition must satisfy $d^2p=Ω(N/L^3)$, where $d$ is the state dimension and $p$ is the per-scalar precision. Conversely, $K$-function composition is solved exactly by a $(K+1)$-layer generalized SSM with $d=1$ and $p=Θ(\log N)$. This gives a worst-case depth hierarchy for this formal problem family. We then distinguish post-input reasoning, in which all thought tokens are generated after the input, from input-interleaved reasoning, in which thought tokens may be inserted while the input stream is being read. Post-input reasoning does not circumvent our communication-based lower-bound pipeline, whereas input-interleaved reasoning admits bidirectional simulations with general deterministic one-pass streaming algorithms at the granularity of persistent memory. Finally, width and precision are not interchangeable under exact step-preserving simulation in the base affine-state model, but become interchangeable through the streaming-memory characterization once input-interleaved reasoning is allowed.
Block-Biased Mamba for Long-Range Sequence Processing
Center for Applied Mathematics(应用数学中心);Cornell University(康奈尔大学);Lawrence Berkeley National Laboratory(伯克利国家实验室);International Computer Science Institute(国际计算机科学研究所)
Deep Direct Discriminative Decoders for High-dimensional Time-series Data Analysis
Krembil Brain Institute, University Health Network(大学健康网络克雷姆比尔脑研究所);KITE Research Institute, Rehabilitation Institute, University Health Network(大学健康网络康复研究所KITE研究院);University of Toronto(多伦多大学);Boston University(波士顿大学);Worcester Polytechnic Institute (WPI)(伍斯特理工学院(WPI))
The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity
The University of Chicago(芝加哥大学);University of New South Wales(新南威尔士大学);The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Simons Institute for the Theory of Computing at UC Berkeley(加州大学伯克利分校西蒙斯理论计算研究所)
Theoretical Foundations of Deep Selective State-Space Models
Imperial College London(伦敦帝国学院);MPI for Intelligent Systems(马克斯·普朗克智能系统研究所);Tübingen AI Center(图宾根人工智能中心);ELLIS Institute Tübingen(ELLIS图宾根研究所);University of Oxford(牛津大学)
Universal Time Series Generation with Neural Controlled Differential Equations
Heidelberg Institute for Theoretical Studies(海德堡理论研究所);IAR, Karlsruhe Institute of Technology(卡尔斯鲁厄技术大学IAR部门);Max Planck Institute for Polymer Research(马克斯·普朗克聚合物研究所);IWR, Heidelberg University(海德堡大学IWR部门);Dept. of Computer Science, University of Toronto(多伦多大学计算机科学系);Mathematical Institute, University of Oxford(牛津大学数学研究所);Vector Institute, Toronto, Canada(多伦多向量研究所)
Center for Foundation Models and Generative AI, Northwestern University(基础模型与生成式AI中心,西北大学);Department of Computer Science, Northwestern University(计算机科学系,西北大学);Ensemble AI;University of California, San Diego(加州大学圣地亚哥分校);Department of Statistics and Data Science, Northwestern University(统计与数据科学系,西北大学)
Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Simons Institute for the Theory of Computing at UC Berkeley(加州大学伯克利分校西蒙斯理论计算研究所);Anhui University(安徽大学)
A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention
Taylor Lab, Huawei(华为泰勒实验室);Key Laboratory of Interdisciplinary Research of Computation and Economics(计算与经济学交叉研究关键实验室);Shanghai University of Finance and Economics(上海财经大学)
DSSMs: State Space Models with Explicit Memory via Delay Differential Equations
College of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院);Department of Mathematics, National University of Singapore(新加坡国立大学数学系);School of Mathematical Sciences, Zhejiang University(浙江大学数学科学学院)
Circuit Complexity Bounds for Visual Autoregressive Model
The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Simons Institute for the Theory of Computing at UC Berkeley(加州大学伯克利分校西蒙斯理论计算研究所)