推进智能序列建模：从S4到Mamba的状态空间架构的演进、权衡与应用
Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
AI总结：
本文综述了状态空间架构从S4到Mamba的演进，分析了其在计算效率、内存优化和推理速度上的创新，并探讨了在自然语言处理、语音识别等领域的超长序列处理优势及面临的挑战。
摘要：
结构化状态空间模型（SSMs）作为一种变革性的序列建模范式，解决了RNN和Transformer的关键限制，如消失梯度、序列计算瓶颈和二次内存复杂度。通过结合结构化递归与状态空间表示，SSMs实现了线性或近线性的计算扩展，擅长长距离依赖任务。本文系统地追溯了SSMs从基础的结构化状态空间序列（S4）模型到现代变种如Mamba、简化结构化状态空间序列（S5）和Jamba的演变，分析了增强计算效率、内存优化和推理速度的架构创新。我们批判性地评估了SSM设计中的权衡，如在表达力与计算约束之间平衡，以及整合混合架构以实现领域特定性能。在自然语言处理、语音识别、计算机视觉和时间序列预测等领域，SSMs在处理超长序列方面表现出最先进的结果，优于基于Transformer的模型在速度和内存利用率方面。案例研究突显了SSMs在实时语音合成和基因序列建模等应用中的优势，其中SSMs将推理延迟降低了传统方法的60%。然而，训练动态、可解释性和硬件感知优化仍存在挑战。本文最后展望了SSMs重新定义可扩展深度学习的潜力，提出了混合系统、理论保证和在资源受限环境中的更广泛应用方向。
英文摘要：
Structured State Space Models (SSMs) have become a prominent class of sequence models, developed against two long-standing difficulties: the sequential computation and gradient propagation limits of Recurrent Neural Networks (RNNs), and the quadratic time and memory cost of self-attention in Transformers. By combining structured recurrence with state-space representations, SSMs attain linear or near-linear scaling in sequence length and hold a constant-size recurrent state during autoregressive decoding, so that no key-value cache grows with context. This paper is a structured review of the lineage running from the Structured State Space Sequence model (S4), through its diagonal and simplified successors S4D, Diagonal State Spaces (DSS) and S5, to the selective models Mamba and Mamba-2, and on to SSM-attention hybrids. Rather than describing models one at a time, the analysis is organized around cross-cutting design dimensions: input-dependent selectivity, the recurrent and convolutional views and when each is preferable, diagonalization, cache behavior at inference, hardware-aware kernels, and hybridization. Structured state-space duality is treated as a central thread, because it accounts for the equivalence between the recurrent and the attention-like form and for why hybrid designs work. Reported efficiency and accuracy results are presented with their measurement configuration and evidence level, because observed speedups reflect the combined effects of the algorithm, implementation kernel and hardware rather than an intrinsic property of the model alone. SSMs are competitive with Transformers and more efficient in specific long-context regimes, whereas attention retains an advantage on tasks dominated by exact retrieval and associative recall. The review closes with limitations, failure modes, deployment considerations and open problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
