arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.06218cs.LGcs.AI
收藏

面向百万级重复建模的长序列模型的统一视图

A Unified View of Long-Sequence Models towards Modeling Million-Scale Dependencies

ETH Zurich(苏黎世联邦理工学院)

Hongyu Hè, Marko Kabic

摘要:

自从它们的概念,Transformers已经接管了传统的序列模型在许多任务,如NLP,图像分类,视频/音频处理,其快速训练和优越的性能。大部分的优点是由于位置编码和多头注意力。然而,Transformers在学习长距离依赖关系方面存在不足,这主要是由于在时间和空间方面,随着上下文长度而缩放的二次复杂度。因此,在过去的五年里,已经提出了无数的方法来提高Transformers的效率。在这项工作中,我们首先后退一步,研究和比较现有的解决方案,以长序列建模的纯数学公式。具体来说,我们使用统一的模板来总结它们,因为它们具有令牌混合的共享性质。通过基准测试,我们证明了长上下文长度确实会产生更好的性能,尽管依赖于应用程序,传统的Transformer模型在利用长距离依赖关系方面存在不足。接下来,受新兴的巨大容量的稀疏模型的启发,我们提出了一个机器学习系统来处理百万级的依赖关系。作为一个概念的证明,我们评估该系统的一个重要组成部分,即分布式多头注意力的性能。我们表明,我们的算法可以扩大注意力计算几乎$40\times$使用四个GeForce RTX 4090 GPU,相比香草多头注意力机制。我们相信这项研究是朝着建立百万级依赖关系模型迈出的重要一步。

英文摘要:

Ever since their conception, Transformers have taken over traditional sequence models in many tasks, such as NLP, image classification, and video/audio processing, for their fast training and superior performance. Much of the merit is attributable to positional encoding and multi-head attention. However, Transformers fall short in learning long-range dependencies mainly due to the quadratic complexity scaled with context length, in terms of both time and space. Consequently, over the past five years, a myriad of methods has been proposed to make Transformers more efficient. In this work, we first take a step back, study and compare existing solutions to long-sequence modeling in terms of their pure mathematical formulation. Specifically, we summarize them using a unified template, given their shared nature of token mixing. Through benchmarks, we then demonstrate that long context length does yield better performance, albeit application-dependent, and traditional Transformer models fall short in taking advantage of long-range dependencies. Next, inspired by emerging sparse models of huge capacity, we propose a machine learning system for handling million-scale dependencies. As a proof of concept, we evaluate the performance of one essential component of this system, namely, the distributed multi-head attention. We show that our algorithm can scale up attention computation by almost $40\times$ using four GeForce RTX 4090 GPUs, compared to vanilla multi-head attention mechanism. We believe this study is an instrumental step towards modeling million-scale dependencies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑