arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.14501cs.LGcs.AIcs.CC
收藏

多层状态空间模型的表达能力与局限性

On the Expressive Power and Limitations of Multi-Layer SSMs

Robotics and Perception Group(机器人与感知组);University of Zurich(苏黎世大学);Shenzhen International Center For Industrial And Applied Mathematics(深圳国际工业与应用数学中心);Shenzhen Research Institute of Big Data(深圳大数据研究 institute);Tengen Intelligence Institute(腾格智能研究所);CRRC Zhuzhou Institute(CRRC 长沙研究所)

Nikola Zubić, Qian Li, Yuyi Wang, Davide Scaramuzza

AI总结:

研究多层状态空间模型的表达能力与限制,揭示其在组合任务中的不足,探讨链式推理对模型表达能力的影响,并分析宽度与精度之间的权衡。

摘要:

我们研究了多层状态空间模型(SSMs)的表达能力和限制。首先,我们证明多层SSMs在组合任务中面临根本限制,揭示了SSMs与流模型之间的固有差距。然后,我们考察了链式推理(CoT)的作用,显示离线CoT不从根本上增加表达能力,而在线CoT能显著提升其能力。事实上,使用在线CoT后,多层SSMs在能力上等同于流算法。最后,我们研究了宽度与精度之间的权衡,证明这些资源在基模型中不可互换,但在允许在线CoT时则存在清晰的等价性。总体而言,我们的结果提供了统一的视角,说明深度、有限精度和CoT如何塑造SSMs的功率和限制。

英文摘要:

We study how depth, finite precision, state dimension, and chain-of-thought (CoT) affect the expressive power of multi-layer state-space models (SSMs). For the explicit-table $K$-function-composition problem, a canonical benchmark for sequential information propagation, we prove that any $L$-layer SSM solving $(L+3)$-function composition must satisfy $d^2p=Ω(N/L^3)$, where $d$ is the state dimension and $p$ is the per-scalar precision. Conversely, $K$-function composition is solved exactly by a $(K+1)$-layer generalized SSM with $d=1$ and $p=Θ(\log N)$. This gives a worst-case depth hierarchy for this formal problem family. We then distinguish post-input reasoning, in which all thought tokens are generated after the input, from input-interleaved reasoning, in which thought tokens may be inserted while the input stream is being read. Post-input reasoning does not circumvent our communication-based lower-bound pipeline, whereas input-interleaved reasoning admits bidirectional simulations with general deterministic one-pass streaming algorithms at the granularity of persistent memory. Finally, width and precision are not interchangeable under exact step-preserving simulation in the base affine-state model, but become interchangeable through the streaming-memory characterization once input-interleaved reasoning is allowed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑