通过分组FIR滤波与注意力汇聚机制增强的结构化状态空间模型
Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
AI总结:
本文提出GFSSM模型,通过分组FIR滤波优化位置编码并引入注意力汇聚机制,缓解Mamba-2的训练困难,提升长序列建模的稳定性与性能。
摘要:
结构化状态空间模型(SSMs)已成为Transformer架构的有力替代方案,具有线性时间复杂度,并在多种序列建模任务中表现出优越性能。尽管具有这些优势,但像原始Mamba-2这样的SSMs由于扩展的循环矩阵乘法序列所引入的敏感性而面临训练困难。在本文中,我们提出了一种先进架构,通过将A乘法分解为多个组,并通过分组有限脉冲响应(FIR)滤波优化位置编码,来缓解这些挑战。这种新结构被称为分组FIR增强SSM(GFSSM),采用半可分离矩阵以实现高效计算。此外,受流式语言模型中发现的“注意力汇聚(attention sink)”现象启发,我们引入了一种类似机制,以增强模型在长序列上的稳定性和性能。我们的方法进一步弥合了SSMs与Transformer架构之间的差距,为可扩展且高性能的序列建模提供了一条可行路径。
英文摘要:
Structured State Space Models (SSMs) have emerged as compelling alternatives to Transformer architectures, offering linear-time complexity and superior performance in various sequence modeling tasks. Despite their advantages, SSMs like the original Mamba-2 face training difficulties due to the sensitivities introduced by the extended series of recurrent matrix multiplications. In this paper, we propose an advanced architecture that mitigates these challenges by decomposing A-multiplications into multiple groups and optimizing positional encoding through Grouped Finite Impulse Response (FIR) filtering. This new structure, denoted as Grouped FIR-enhanced SSM (GFSSM), employs semiseparable matrices for efficient computation. Furthermore, inspired by the "attention sink" phenomenon identified in streaming language models, we incorporate a similar mechanism to enhance the stability and performance of our model over extended sequences. Our approach further bridges the gap between SSMs and Transformer architectures, offering a viable path forward for scalable and high-performing sequence modeling.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。