arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.01925cs.LG
收藏

学习用于 Transformer 中线性相对位置编码的傅里叶变换

Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers

Google Research(谷歌研究院);Columbia University(哥伦比亚大学);Carnegie Mellon University(卡内基梅隆大学);Waymo(Waymo公司);Peking University(北京大学);University of Cambridge(剑桥大学);Alan Turing Institute(阿兰·图灵研究所)

Krzysztof Marcin Choromanski, Shanda Li, Valerii Likhosherstov, Kumar Avinava Dubey, Shengjie Luo, Di He, Yiming Yang, Tamas Sarlos, Thomas Weingarten, Adrian Weller

AI总结:

本文提出 FourierLearner-Transformers,通过学习 RPE 的谱表示隐式构造最优相对位置编码,在保持线性注意力与低内存开销的同时支持序列、图像和三维分子等任务。

摘要:

我们提出了一类名为 FourierLearner-Transformers(FLT)的新型线性 Transformer,它融合了广泛的相对位置编码机制(RPE)。其中既包括应用于序列数据的常规 RPE 技术,也包括作用于嵌入在高维欧几里得空间中的几何数据的新型 RPE。FLT 通过学习最优 RPE 机制的谱表示来隐式构建该机制。与其他将高效低秩线性注意力与 RPE 相结合的架构不同,FLT 在内存使用方面保持实用性,并且不需要对 RPE 掩码的结构作出额外假设。此外,FLT 允许应用某些结构性归纳偏置技术来指定掩码策略;例如,它们提供了一种学习本文所引入的所谓局部 RPE 的方法,并且与用于语言建模的其他若干线性 Transformer 相比能够提升准确率。我们还在其他数据模态和任务上全面测试了 FLT,例如图像分类、三维分子建模以及可学习优化器。据我们所知,对于三维分子数据,FLT 是首个提供线性注意力并融合 RPE 掩码的 Transformer 架构。

英文摘要:

We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs). These include regular RPE techniques applied for sequential data, as well as novel RPEs operating on geometric data embedded in higher-dimensional Euclidean spaces. FLTs construct the optimal RPE mechanism implicitly by learning its spectral representation. As opposed to other architectures combining efficient low-rank linear attention with RPEs, FLTs remain practical in terms of their memory usage and do not require additional assumptions about the structure of the RPE mask. Besides, FLTs allow for applying certain structural inductive bias techniques to specify masking strategies, e.g. they provide a way to learn the so-called local RPEs introduced in this paper and give accuracy gains as compared with several other linear Transformers for language modeling. We also thoroughly test FLTs on other data modalities and tasks, such as image classification, 3D molecular modeling, and learnable optimizers. To the best of our knowledge, for 3D molecular data, FLTs are the first Transformer architectures providing linear attention and incorporating RPE masking.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑