arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.16620cs.LGcs.CL
收藏

关于Transformer的长程能力

On the Long Range Abilities of Transformers

The Blavatnik School of Computer Science(布拉瓦尼克计算机科学学院);Tel Aviv University(特拉维夫大学)

Itamar Zimerman, Lior Wolf

AI总结:

本文通过向Transformer注意力机制引入平滑性归纳偏置和局部性,以极小的计算代价且不增加可训练参数,显著提升了其在Long Range Arena基准上的长程任务表现,并揭示了Transformer长程能力不足的原因。

摘要:

尽管Transformer架构在现代深度学习,尤其是自然语言处理领域占据主导地位,但与近期专门为长程任务设计的层相比,其在长程任务上表现出次优性能。在本工作中,受长程层关键属性的启发,如状态空间层、线性RNN层和全局卷积层,我们证明了对Transformer架构进行最小限度的修改即可显著提升其在Long Range Arena(LRA)基准上的性能,从而缩小与这些专用层的差距。我们确定了长程任务的两个关键原则:(i)引入对平滑性的归纳偏置,以及(ii)局部性。正如我们所展示的,将这些思想融入注意力机制可以改善结果,且额外的计算量可忽略不计,同时无需增加任何可训练参数。我们的理论和实验还揭示了Transformer在长程任务上性能较差的原因,并确定了成功捕获长程依赖所必需的关键性质。

英文摘要:

Despite their dominance in modern DL and, especially, NLP domains, transformer architectures exhibit sub-optimal performance on long-range tasks compared to recent layers that are specifically designed for this purpose. In this work, drawing inspiration from key attributes of long-range layers, such as state-space layers, linear RNN layers, and global convolution layers, we demonstrate that minimal modifications to the transformer architecture can significantly enhance performance on the Long Range Arena (LRA) benchmark, thus narrowing the gap with these specialized layers. We identify that two key principles for long-range tasks are (i) incorporating an inductive bias towards smoothness, and (ii) locality. As we show, integrating these ideas into the attention mechanism improves results with a negligible amount of additional computation and without any additional trainable parameters. Our theory and experiments also shed light on the reasons for the inferior performance of transformers on long-range tasks and identify critical properties that are essential for successfully capturing long-range dependencies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑