分数阶状态空间转换用于长序列建模
Fractional State Space Transition for Long Sequence Modeling
浏览论文内容
中文总结 AI 辅助
本文提出FRAC，一种基于分数阶动力学的选择性状态空间模型，用幂律长记忆替代指数遗忘，通过有限状态对数间隔指数模式近似实现高效递归，实验证明其在长上下文建模中优于现有SSM基线。
中文摘要 AI 辅助
状态空间模型（SSMs）将序列历史压缩为有界递归状态，使得由此产生的记忆法则成为影响长上下文性能的关键架构选择。大多数现代SSMs依赖于基于常微分方程（ODE）的动力学，这导致指数遗忘，限制了其在广泛时间范围内保留信息的能力。我们提出了FRAC，一种源自分数阶动力学的选择性SSM架构，用幂律长记忆取代了这种指数衰减。为了使分数阶动力学实用化，FRAC用有限状态、对数间隔的指数模式之和来近似重尾目标核。这种构造将分数阶记忆转化为高效的递归模块，支持并行训练和预填充，同时保持有界状态的自回归解码。大量实验，包括13亿参数的语言建模，表明FRAC在长上下文性能上持续优于最先进的SSM基线，同时在短上下文上保持竞争力。这些结果表明，分数阶动力学为长上下文SSMs提供了一种实用且有效的先验。
英文摘要
State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges. We introduce FRAC, a selective SSM architecture derived from fractional dynamics that replaces this exponential decay with power-law long memory. To make fractional dynamics practical, FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes. This construction turns fractional memory into an efficient recurrent module with parallel training and prefill, while retaining bounded-state autoregressive decoding. Extensive experiments, including 1.3B-parameter language modeling, demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines while staying competitive on short-context. These results show that fractional dynamics provide a practical and effective prior for long-context SSMs.
