arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.08136cs.CLcs.LG
收藏

基于状态空间增广Transformer的长序列建模

Efficient Long Sequence Modeling via State Space Augmented Transformer

Georgia Institute of Technology(佐治亚理工学院);Microsoft(微软)

Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, Jianfeng Gao

摘要:

Transformer模型在各种自然语言处理任务中取得了卓越的性能。然而,注意力机制的二次计算成本限制了其对长序列的实用性。现有的注意力变体提高了计算效率,但它们有效计算全局信息的能力有限。与Transformer模型并行,状态空间模型(SSM)是为长序列定制的,但是它们不够灵活以捕获复杂的局部信息。我们提出了SPADE,简称$\underline{\textbf{S}}$状态$\underline{\textbf{P}}$ ace $\underline{\textbf{A}}$ ugmente $\underline{\textbf{D}}$ Transform $\underline{\textbf{E}}$ r。具体来说,我们增加了SSM到SPADE的底层,我们采用有效的本地注意力方法的其他层。SSM增强了全局信息,这补充了局部注意力方法中缺乏长距离依赖性问题。在Long Range Arena基准测试和语言建模任务上的实验结果证明了该方法的有效性。为了进一步证明SPADE的可扩展性,我们对大型编码器-解码器模型进行了预训练,并在自然语言理解和自然语言生成任务上呈现了微调结果。

英文摘要:

Transformer models have achieved superior performance in various natural language processing tasks. However, the quadratic computational cost of the attention mechanism limits its practicality for long sequences. There are existing attention variants that improve the computational efficiency, but they have limited ability to effectively compute global information. In parallel to Transformer models, state space models (SSMs) are tailored for long sequences, but they are not flexible enough to capture complicated local information. We propose SPADE, short for $\underline{\textbf{S}}$tate s$\underline{\textbf{P}}$ace $\underline{\textbf{A}}$ugmente$\underline{\textbf{D}}$ Transform$\underline{\textbf{E}}$r. Specifically, we augment a SSM into the bottom layer of SPADE, and we employ efficient local attention methods for the other layers. The SSM augments global information, which complements the lack of long-range dependency issue in local attention methods. Experimental results on the Long Range Arena benchmark and language modeling tasks demonstrate the effectiveness of the proposed method. To further demonstrate the scalability of SPADE, we pre-train large encoder-decoder models and present fine-tuning results on natural language understanding and natural language generation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑