序列建模的一个核心挑战是高效处理具有扩展上下文的任务。尽管最近的状态空间模型(SSMs)在这一领域取得了显著进展,但它们往往缺乏输入依赖的滤波能力,或者需要大幅增加模型复杂度来处理输入变化。我们通过引入 S7 来弥补这一不足,S7 是一种简化但功能强大的 SSM,能够处理输入依赖性,同时结合稳定的重参数化和特定的设计选择,根据输入内容动态调整状态转移,从而保持效率和性能。我们证明,这种重参数化通过使状态转移随时间保持良好行为,确保了长序列建模的稳定性。此外,它还控制了梯度范数,从而实现高效训练,并防止梯度爆炸或梯度消失等问题。S7 在各种序列建模任务中显著优于基线,包括神经形态事件数据集、Long Range Arena 基准测试以及各种物理和生物时间序列。总体而言,S7 提供了一种更直接的序列建模方法,无需依赖复杂的、领域特定的归纳偏置,在关键基准测试中取得了显著改进。
英文摘要:
A central challenge in sequence modeling is efficiently handling tasks with extended contexts. While recent state-space models (SSMs) have made significant progress in this area, they often lack input-dependent filtering or require substantial increases in model complexity to handle input variability. We address this gap by introducing S7, a simplified yet powerful SSM that can handle input dependence while incorporating stable reparameterization and specific design choices to dynamically adjust state transitions based on input content, maintaining efficiency and performance. We prove that this reparameterization ensures stability in long-sequence modeling by keeping state transitions well-behaved over time. Additionally, it controls the gradient norm, enabling efficient training and preventing issues like exploding or vanishing gradients. S7 significantly outperforms baselines across various sequence modeling tasks, including neuromorphic event-based datasets, Long Range Arena benchmarks, and various physical and biological time series. Overall, S7 offers a more straightforward approach to sequence modeling without relying on complex, domain-specific inductive biases, achieving significant improvements across key benchmarks.
Simplified State Space Layers for Sequence Modeling
Institute for Computational and Mathematical Engineering, Stanford University(斯坦福大学计算与数学工程研究所);Wu Tsai Neurosciences Institute, Stanford University(斯坦福大学吴蔡神经科学研究所);Department of Statistics, Stanford University(斯坦福大学统计学系)
ELLIS Institute Tübingen(图宾根埃利斯研究所);LIONS, EPFL(日内瓦理工学院LIONS实验室);University of Freiburg(弗赖堡大学);Max-Planck-Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);Prior Labs(Prior实验室)
Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces
University of Toronto(多伦多大学);Vector Institute for Artificial Intelligence(矢量人工智能研究所);University Health Network(大学网络健康中心);Peter Munk Cardiac Centre(彼得·芒克心脏中心);Department of Laboratory Medicine and Pathobiology, University of Toronto(多伦多大学检验医学与病理生物学系);AI Hub, University Health Network(大学网络健康中心人工智能中心)
SSUMamba: Spatial-Spectral Selective State Space Model for Hyperspectral Image Denoising
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);School of Information and Communication Technology, Griffith University(格里菲斯大学信息与通信技术学院)
该研究提出 SSUMamba,通过六种空间-光谱连续扫描顺序、双向 SSM 与 3D 卷积高效建模高光谱图像的长程及局部依赖,在降低每批内存消耗的同时提升去噪性能。
Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Institute of High Performance Computing, A*STAR(新加坡高性能计算研究院(新加坡科技研究局));Lehigh University(理海大学)
MambaLRP: Explaining Selective State Space Sequence Models
Technische Universität Berlin(柏林工业大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所);Freie Universität Berlin(柏林自由大学);Korea University(高丽大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);Google DeepMind(谷歌DeepMind)