arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.16249cs.LGcs.AI
收藏

用于高效双向序列建模的线性注意力

Linear Attention for Efficient Bidirectional Sequence Modeling

Arshia Afzal, Elias Abad Rocamora, Leyla Naz Candogan, Pol Puigdemont, Francesco Tonin, Yongtao Wu, Mahsa Shoaran, Volkan Cevher

AI总结:

针对线性Transformer缺乏双向序列建模统一框架的问题,提出LION框架将三类核心表示扩展到双向场景,证明其理论等价性,通过多衰减类型变体验证其性能可匹配或超越softmax Transformer,且训练推理效率优于现有状态空间模型。

摘要:

线性Transformer(Linear Transformer)和状态空间模型(State Space Model)已成为因果序列建模中softmax Transformer的高效替代方案,可通过矩阵乘法实现并行训练,并支持高效的RNN风格推理。然而,尽管它们在因果任务中取得了成功,目前仍不存在将线性Transformer应用于双向序列建模的统一框架。我们提出了LION,这是首个系统性地将线性Transformer扩展到双向场景的框架。LION将因果场景中常用的三种核心表示——全量线性注意力(full Linear Attention)、双向RNN以及分块并行形式——推广到了双向场景。这些形式在理论上是等价的,能让模型在训练和推理过程中分别利用各自的优势。我们证明了一大类线性Transformer都可以通过LION进行扩展,并基于衰减类型的选择,通过三个核心示例验证了我们的框架:LION-LIT,即arXiv:2006.16236工作的双向扩展版本;LION-D,基于arXiv:2307.08621的工作;以及LION-S,一种使用选择性衰减的变体,相关工作见arXiv:2103.02143、arXiv:2312.0075。在标准双向任务中,LION使模型的性能能够匹配或超越softmax Transformer,同时相比现有状态空间模型具备显著更快的训练速度和更高效的推理效率。

英文摘要:

Linear Transformers and State Space Models have emerged as efficient alternatives to softmax Transformers for causal sequence modeling, enabling parallel training via matrix multiplication and efficient RNN-style inference. However, despite their success in causal tasks, no unified framework exists for applying Linear Transformers to bidirectional sequence modeling. We introduce LION, the first framework to systematically extend Linear Transformers to the bidirectional setting. LION generalizes three core representations commonly used in the causal case - full Linear Attention , bidirectional RNN, and chunkwise parallel form - to the bidirectional setting. These forms are theoretically equivalent and enable models to exploit the strengths of each during training and inference. We prove that a broad class of Linear Transformers can be extended using LION and validate our framework via three core examples based on the choice of decay type: LION-LIT, the bidirectional extension of arXiv:2006.16236; LION-D, based on arXiv:2307.08621; and LION-S, a variant using selective decay arXiv:2103.02143, arXiv:2312.0075. Across standard bidirectional tasks, LION enables models to match or exceed the performance of softmax Transformers, while offering significantly faster training and more efficient inference than existing State Space Models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑