Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers
Google Research(谷歌研究院);Columbia University(哥伦比亚大学);Carnegie Mellon University(卡内基梅隆大学);Waymo(Waymo公司);Peking University(北京大学);University of Cambridge(剑桥大学);Alan Turing Institute(阿兰·图灵研究所)
Krzysztof Marcin Choromanski, Shanda Li, Valerii Likhosherstov, Kumar Avinava Dubey, Shengjie Luo, Di He, Yiming Yang, Tamas Sarlos, Thomas Weingarten, Adrian Weller
We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs). These include regular RPE techniques applied for sequential data, as well as novel RPEs operating on geometric data embedded in higher-dimensional Euclidean spaces. FLTs construct the optimal RPE mechanism implicitly by learning its spectral representation. As opposed to other architectures combining efficient low-rank linear attention with RPEs, FLTs remain practical in terms of their memory usage and do not require additional assumptions about the structure of the RPE mask. Besides, FLTs allow for applying certain structural inductive bias techniques to specify masking strategies, e.g. they provide a way to learn the so-called local RPEs introduced in this paper and give accuracy gains as compared with several other linear Transformers for language modeling. We also thoroughly test FLTs on other data modalities and tasks, such as image classification, 3D molecular modeling, and learnable optimizers. To the best of our knowledge, for 3D molecular data, FLTs are the first Transformer architectures providing linear attention and incorporating RPE masking.
Deconstructing Positional Information: From Attention Logits to Training Biases
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Antai College of Economics and Management, Shanghai Jiao Tong University(上海交通大学安泰经济管理学院)
Alternative positional encoding functions for neural transformers
Department of Computer Languages and Computer Science University of Málaga(计算机语言与计算机科学系马拉加大学);Department of Systems Engineering and Automation University of Málaga(系统工程与自动化系马拉加大学);ITIS Software. Universidad de Málaga(ITIS软件. 马拉加大学)
Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
MALTA – Machine Learning Theory and Applications Lab, PUCRS – Porto Alegre, Brazil(MALTA机器学习理论与应用实验室,PUCRS-波士顿实验室,巴西);Kunumi Institute, Brazil(昆米研究所,巴西)
Structural Positional Encoding for knowledge integration in transformer-based medical process monitoring
Laboratorio Integrato di Intelligenza Artificiale e Informatica in Medicina DAIRI(人工智能与医学信息整合实验室DAIRI);Azienda Ospedaliera SS. Antonio e Biagio e Cesare Arrigo(安东尼和比阿乔和卡雷罗医院)
GIN-SD: Source Detection in Graphs with Incomplete Nodes via Positional Encoding and Attentive Fusion
School of Computer Science, Northwestern Polytechnical University (NWPU)(西北工业大学计算机学院);School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University (NWPU)(西北工业大学光电与智能研究院);Cyberspace Institute of Advanced Technology, Guangzhou University(广州大学网络空间先进技术研究院)
Generalized Laplacian Positional Encoding for Graph Representation Learning
Ludwig-Maximilian University of Munich(慕尼黑大学);MILA(蒙特利尔学习算法研究所);New Jersey Institute of Technology(新泽西理工学院);TU Wien(维也纳工业大学);Massachusetts Institute of Technology(麻省理工学院);University of Vienna(维也纳大学);NVIDIA Research(英伟达研究院)
Remember to Forget: Gated Adaptive Positional Encoding
Department of Computer Science and Technology, University of Cambridge(计算机科学与技术系,剑桥大学);Department of Computer, Control and Management Engineering, Sapienza University(计算机、控制与管理工程系,萨皮恩扎大学);Department of Information Engineering, University of Padova(信息工程系,帕多瓦大学)
What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding
Rensselaer Polytechnic Institute(伦斯勒理工学院);Stony Brook University(石溪大学);Michigan State University(密歇根州立大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);IBM Research(IBM研究院);University of Science and Technology of China(中国科学技术大学);IBM Thomas J. Watson Research Center(IBM托马斯·J.沃森研究中心)
Graph Positional Encoding via Random Feature Propagation
Ben-Gurion University of the Negev(内盖夫本-古里安大学);Imperial College London(伦敦帝国学院);Purdue University(普渡大学);Bar-Ilan University(巴伊兰大学);NVIDIA Research(英伟达研究院)
School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);Hopcroft Center on Computing Science, Huazhong University of Science and Technology(华中科技大学计算科学霍普克罗夫特中心);Department of Computer Science, Cornell University(康奈尔大学计算机科学系)
LEAP: Local ECT-Based Learnable Positional Encodings for Graphs
AIDOS Lab, University of Fribourg(弗里堡大学AIDOS实验室);Institute of AI for Health, Helmholtz Munich(慕尼黑亥姆霍兹人工智能与健康研究所);Technical University of Munich(慕尼黑技术大学);Department of Computer Science, ETH Zurich(苏黎世联邦理工学院计算机科学系);Department of Computer Science, University of Basel(巴塞尔大学计算机科学系)
Do traveling waves make good positional encodings?
Institute of Computer Science and Campus Institute Data Science, University of Göttingen(哥廷根大学计算机科学学院和校园数据科学研究所);Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany(马克斯·普朗克动态与自组织研究所)