DSSMs：通过延迟微分方程实现的具有显式记忆的状态空间模型

中文摘要 AI 辅助

状态空间模型（SSMs）已成为高效长序列建模的强大范式，提供并行训练和快速线性时间循环推理。然而，与其他循环架构一样，SSMs必须将无界历史压缩到固定大小的状态中，这限制了上下文保留，并使得对长距离上下文进行精确检索本质上变得困难。为了克服这一限制，我们提出了延迟状态空间模型（DSSMs），这是一种受延迟微分方程（DDE）启发的对角SSMs扩展，它通过显式延迟状态反馈增强离散SSM循环。使显式延迟反馈实用化需要新的稳定性参数化、历史管理和FFT训练工具。我们通过基于简单的与延迟无关的稳定性条件进行实际离散化和参数化来应对这些挑战。为了绕过直接的时域内核构建，我们推导了DSSM传递函数并在频域中计算内核，使用内核轮廓移位来抑制混叠并恢复准确的FFT训练。从经验上看，DSSMs在目标延迟检索任务上有显著改进，同时在大多数标准序列指标上优于S4D，在其他指标上与S4D相近。

英文摘要

State Space Models (SSMs) have emerged as a powerful paradigm for efficient long-sequence modeling, offering parallel training and fast linear-time recurrent inference. However, like other recurrent architectures, SSMs must compress an unbounded history into a fixed-size state, which limits context retention and makes precise retrieval over long-range context inherently difficult. To overcome this limitation, we propose Delay State Space Models (DSSMs), a delay differential equation (DDE)-inspired extension of diagonal SSMs that augments discrete SSM recurrences with explicit delayed-state feedback. Making explicit delayed feedback practical requires new stability parameterization, history management, and FFT-training tools. We address these challenges with a practical discretization and parameterization grounded in a simple delay-independent stability condition. To bypass direct time-domain kernel construction, we derive the DSSM transfer function and compute kernels in the frequency domain, using a kernel contour shift to suppress aliasing and recover accurate FFT training. Empirically, DSSMs substantially improve targeted delayed-retrieval tasks while outperforming S4D on most standard sequence metrics and remaining close on the others.

发表机构

• College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）
• Department of Mathematics, National University of Singapore（新加坡国立大学数学系）
• School of Mathematical Sciences, Zhejiang University（浙江大学数学科学学院）

机构由 AI 辅助整理，请以论文原文为准。