Transformers achieve remarkable performance in various domains, including NLP, CV, audio processing, and graph analysis. However, they do not scale well on long sequence tasks due to their quadratic complexity w.r.t. the inputs length. Linear Transformers were proposed to address this limitation. However, these models have shown weaker performance on the long sequence tasks comparing to the original one. In this paper, we explore Linear Transformer models, rethinking their two core components. Firstly, we improved Linear Transformer with Shift-Invariant Kernel Function SIKF, which achieve higher accuracy without loss in speed. Secondly, we introduce FastRPB which stands for Fast Relative Positional Bias, which efficiently adds positional information to self-attention using Fast Fourier Transformation. FastRPB is independent of the self-attention mechanism and can be combined with an original self-attention and all its efficient variants. FastRPB has O(N log(N)) computational complexity, requiring O(N) memory w.r.t. input sequence length N.
Deconstructing Positional Information: From Attention Logits to Training Biases
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Antai College of Economics and Management, Shanghai Jiao Tong University（上海交通大学安泰经济管理学院）
Alternative positional encoding functions for neural transformers
Department of Computer Languages and Computer Science University of Málaga（计算机语言与计算机科学系马拉加大学）；Department of Systems Engineering and Automation University of Málaga（系统工程与自动化系马拉加大学）；ITIS Software. Universidad de Málaga（ITIS软件. 马拉加大学）
Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
MALTA – Machine Learning Theory and Applications Lab, PUCRS – Porto Alegre, Brazil（MALTA机器学习理论与应用实验室，PUCRS-波士顿实验室，巴西）；Kunumi Institute, Brazil（昆米研究所，巴西）
Structural Positional Encoding for knowledge integration in transformer-based medical process monitoring
Laboratorio Integrato di Intelligenza Artificiale e Informatica in Medicina DAIRI（人工智能与医学信息整合实验室DAIRI）；Azienda Ospedaliera SS. Antonio e Biagio e Cesare Arrigo（安东尼和比阿乔和卡雷罗医院）
GIN-SD: Source Detection in Graphs with Incomplete Nodes via Positional Encoding and Attentive Fusion
School of Computer Science, Northwestern Polytechnical University (NWPU)（西北工业大学计算机学院）；School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University (NWPU)（西北工业大学光电与智能研究院）；Cyberspace Institute of Advanced Technology, Guangzhou University（广州大学网络空间先进技术研究院）
Generalized Laplacian Positional Encoding for Graph Representation Learning
Ludwig-Maximilian University of Munich（慕尼黑大学）；MILA（蒙特利尔学习算法研究所）；New Jersey Institute of Technology（新泽西理工学院）；TU Wien（维也纳工业大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Vienna（维也纳大学）；NVIDIA Research（英伟达研究院）
Remember to Forget: Gated Adaptive Positional Encoding
Department of Computer Science and Technology, University of Cambridge（计算机科学与技术系，剑桥大学）；Department of Computer, Control and Management Engineering, Sapienza University（计算机、控制与管理工程系，萨皮恩扎大学）；Department of Information Engineering, University of Padova（信息工程系，帕多瓦大学）
What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Stony Brook University（石溪大学）；Michigan State University（密歇根州立大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；University of Science and Technology of China（中国科学技术大学）；IBM Thomas J. Watson Research Center（IBM托马斯·J.沃森研究中心）
Graph Positional Encoding via Random Feature Propagation
Ben-Gurion University of the Negev（内盖夫本-古里安大学）；Imperial College London（伦敦帝国学院）；Purdue University（普渡大学）；Bar-Ilan University（巴伊兰大学）；NVIDIA Research（英伟达研究院）
School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；Hopcroft Center on Computing Science, Huazhong University of Science and Technology（华中科技大学计算科学霍普克罗夫特中心）；Department of Computer Science, Cornell University（康奈尔大学计算机科学系）
LEAP: Local ECT-Based Learnable Positional Encodings for Graphs
AIDOS Lab, University of Fribourg（弗里堡大学AIDOS实验室）；Institute of AI for Health, Helmholtz Munich（慕尼黑亥姆霍兹人工智能与健康研究所）；Technical University of Munich（慕尼黑技术大学）；Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；Department of Computer Science, University of Basel（巴塞尔大学计算机科学系）
Do traveling waves make good positional encodings?
Institute of Computer Science and Campus Institute Data Science, University of Göttingen（哥廷根大学计算机科学学院和校园数据科学研究所）；Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany（马克斯·普朗克动态与自组织研究所）
Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Hierarchical Structures
FPT Software AI Center（FPT软件人工智能中心）；Halıcıoğlu Data Science Institute, University of California San Diego（加州大学圣地亚哥分校哈利哲奥卢数据科学研究所）；University of Chicago（芝加哥大学）
Causality-Induced Positional Encoding for Transformer-Based Representation Learning of Non-Sequential Features
Department of Computer Science, Emory University（埃默里大学计算机科学系）；School of Statistics and Mathematics, Zhongnan University of Economics and Law（中南财经政法大学统计与数学学院）；School of Medicine, Department of Human Genetics, Emory University（埃默里大学医学院人类遗传学系）
