arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.03464cs.LGeess.SPmath.DS
收藏

S7:用于序列建模的选择性与简化状态空间层

S7: Selective and Simplified State Space Layers for Sequence Modeling

University of Zurich(苏黎世大学);ETH Zurich(苏黎世联邦理工学院)

Taylan Soydan, Nikola Zubić, Nico Messikommer, Siddhartha Mishra, Davide Scaramuzza

AI总结:

本文提出简化状态空间模型 S7,通过稳定重参数化实现输入依赖的动态状态转移,在长序列建模中保证稳定性并控制梯度,在多个基准上显著超越基线。

摘要:

序列建模的一个核心挑战是高效处理具有扩展上下文的任务。尽管最近的状态空间模型(SSMs)在这一领域取得了显著进展,但它们往往缺乏输入依赖的滤波能力,或者需要大幅增加模型复杂度来处理输入变化。我们通过引入 S7 来弥补这一不足,S7 是一种简化但功能强大的 SSM,能够处理输入依赖性,同时结合稳定的重参数化和特定的设计选择,根据输入内容动态调整状态转移,从而保持效率和性能。我们证明,这种重参数化通过使状态转移随时间保持良好行为,确保了长序列建模的稳定性。此外,它还控制了梯度范数,从而实现高效训练,并防止梯度爆炸或梯度消失等问题。S7 在各种序列建模任务中显著优于基线,包括神经形态事件数据集、Long Range Arena 基准测试以及各种物理和生物时间序列。总体而言,S7 提供了一种更直接的序列建模方法,无需依赖复杂的、领域特定的归纳偏置,在关键基准测试中取得了显著改进。

英文摘要:

A central challenge in sequence modeling is efficiently handling tasks with extended contexts. While recent state-space models (SSMs) have made significant progress in this area, they often lack input-dependent filtering or require substantial increases in model complexity to handle input variability. We address this gap by introducing S7, a simplified yet powerful SSM that can handle input dependence while incorporating stable reparameterization and specific design choices to dynamically adjust state transitions based on input content, maintaining efficiency and performance. We prove that this reparameterization ensures stability in long-sequence modeling by keeping state transitions well-behaved over time. Additionally, it controls the gradient norm, enabling efficient training and preventing issues like exploding or vanishing gradients. S7 significantly outperforms baselines across various sequence modeling tasks, including neuromorphic event-based datasets, Long Range Arena benchmarks, and various physical and biological time series. Overall, S7 offers a more straightforward approach to sequence modeling without relying on complex, domain-specific inductive biases, achieving significant improvements across key benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑