We propose a novel positional encoding for learning graph on Transformer architecture. Existing approaches either linearize a graph to encode absolute position in the sequence of nodes, or encode relative position with another node using bias terms. The former loses preciseness of relative position from linearization, while the latter loses a tight integration of node-edge and node-topology interaction. To overcome the weakness of the previous approaches, our method encodes a graph without linearization and considers both node-topology and node-edge interaction. We name our method Graph Relative Positional Encoding dedicated to graph representation learning. Experiments conducted on various graph datasets show that the proposed method outperforms previous approaches significantly. Our code is publicly available at https://github.com/lenscloth/GRPE.
On the Stability of Expressive Positional Encodings for Graphs
Georgia Institute of Technology（佐治亚理工学院）；Purdue University（普渡大学）；Stanford University（斯坦福大学）；Tongji University（同济大学）；Peking University（北京大学）；MIT（麻省理工学院）
Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University（南洋理工大学物理与数学科学学院数学科学部）；School of Computing, University of Kent（肯特大学计算机学院）；Department of Computer Science and Technology, University of Cambridge（剑桥大学计算机科学与技术系）
Ludwig-Maximilian University of Munich（慕尼黑大学）；MILA（蒙特利尔学习算法研究所）；New Jersey Institute of Technology（新泽西理工学院）；TU Wien（维也纳工业大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Vienna（维也纳大学）；NVIDIA Research（英伟达研究院）
Graph neural networks (GNNs) are the primary tool for processing graph-structured data. Unfortunately, the most commonly used GNNs, called Message Passing Neural Networks (MPNNs) suffer from several fundamental limitations. To overcome these limitations, recent works have adapted the idea of positional encodings to graph data. This paper draws inspiration from the recent success of Laplacian-based positional encoding and defines a novel family of positional encoding schemes for graphs. We accomplish this by generalizing the optimization problem that defines the Laplace embedding to more genera…
Beijing University of Posts and Telecommunications（北京邮电大学）；China Telecom Research Institute（中国电信研究院）
Learning node representations that incorporate information from graph structure benefits wide range of tasks on graph. The majority of existing graph neural networks (GNNs) have limited power in capturing position information for a given node. The idea of positioning nodes with selected anchors has been exploited, yet mainly relying on explicit labeling of distance information. Here we propose Graph Inference Representation (GIR), an anchor based GNN model encoding path information related to pre-selected anchors for each node. Abilities to get position-aware embeddings are theoretically and e…
Graph Positional Encoding via Random Feature Propagation
Ben-Gurion University of the Negev（内盖夫本-古里安大学）；Imperial College London（伦敦帝国学院）；Purdue University（普渡大学）；Bar-Ilan University（巴伊兰大学）；NVIDIA Research（英伟达研究院）
School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；Hopcroft Center on Computing Science, Huazhong University of Science and Technology（华中科技大学计算科学霍普克罗夫特中心）；Department of Computer Science, Cornell University（康奈尔大学计算机科学系）
Graph neural networks (GNN) have shown great advantages in many graph-based learning tasks but often fail to predict accurately for a task-based on sets of nodes such as link/motif prediction and so on. Many works have recently proposed to address this problem by using random node features or node distance features. However, they suffer from either slow convergence, inaccurate prediction, or high complexity. In this work, we revisit GNNs that allow using positional features of nodes given by positional encoding (PE) techniques such as Laplacian Eigenmap, Deepwalk, etc. GNNs with PE often get c…
LEAP: Local ECT-Based Learnable Positional Encodings for Graphs
AIDOS Lab, University of Fribourg（弗里堡大学AIDOS实验室）；Institute of AI for Health, Helmholtz Munich（慕尼黑亥姆霍兹人工智能与健康研究所）；Technical University of Munich（慕尼黑技术大学）；Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；Department of Computer Science, University of Basel（巴塞尔大学计算机科学系）
Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Hierarchical Structures
FPT Software AI Center（FPT软件人工智能中心）；Halıcıoğlu Data Science Institute, University of California San Diego（加州大学圣地亚哥分校哈利哲奥卢数据科学研究所）；University of Chicago（芝加哥大学）
Alternative positional encoding functions for neural transformers
Department of Computer Languages and Computer Science University of Málaga（计算机语言与计算机科学系马拉加大学）；Department of Systems Engineering and Automation University of Málaga（系统工程与自动化系马拉加大学）；ITIS Software. Universidad de Málaga（ITIS软件. 马拉加大学）
Do traveling waves make good positional encodings?
Institute of Computer Science and Campus Institute Data Science, University of Göttingen（哥廷根大学计算机科学学院和校园数据科学研究所）；Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany（马克斯·普朗克动态与自组织研究所）
Tel Aviv University（特拉维夫大学）；University of Washington（华盛顿大学）；Intel Labs（英特尔实验室）；Meta AI
Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings. However, we show that LMs without any explicit positional encoding are still competitive with standard models, and that this phenomenon is robust across different datasets, model sizes, and sequence lengths. Probing experiments reveal that such models acquire an implicit notion of absolute positions throughout the network, effectively compensating for the missing information. We conjecture that causal attention enables the model to infer the number of pred…
Meituan Inc.（美团公司）；Beijing Academy of AI（北京人工智能研究院）；Zhejiang University（浙江大学）
We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike previous fixed or learnable positional encodings, which are pre-defined and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during training. Besides, CPE can keep the desired translation-invariance in the image classification task, resulting in improved performance. We implement CPE with a simple Position Encoding Generator…
Google Research（谷歌研究院）；UCLA（加州大学洛杉矶分校）；Apple（苹果公司）
Attentional mechanisms are order-invariant. Positional encoding is a crucial component to allow attention-based deep model architectures such as Transformer to address sequences or images where the position of information matters. In this paper, we propose a novel positional encoding method based on learnable Fourier features. Instead of hard-coding each position as a token or a vector, we represent each position, which can be multi-dimensional, as a trainable encoding based on learnable Fourier feature mapping, modulated with a multi-layer perceptron. The representation is particularly advant…
Center for Information and Language Processing（信息与语言处理中心）；Ludwig Maximilian University of Munich（慕尼黑大学）
This paper describes how to apply self-attention with relative positional encodings to the task of relation extraction. We propose to use the self-attention encoder layer together with an additional position-aware attention layer that takes into account positions of the query and the object in the sentence. The self-attention encoder also uses a custom implementation of relative positional encodings which allow each word in the sentence to take into account its left and right context. The evaluation of the model is done on the TACRED dataset. The proposed model relies only on attention (no rec…
Aalto University（阿尔托大学）；University of Bologna（博洛尼亚大学）；University of Gothenburg（哥德堡大学）；Chalmers University of Technology（查尔姆斯理工大学）；YaiYai Ltd（YaiYai有限公司）
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
National University of Singapore（新加坡国立大学）；Center for AI Research, VinUniversity（Vin大学人工智能研究中心）；Independent Researcher（独立研究者）；Technical University of Munich（慕尼黑技术大学）
Structural Positional Encoding for knowledge integration in transformer-based medical process monitoring
Laboratorio Integrato di Intelligenza Artificiale e Informatica in Medicina DAIRI（人工智能与医学信息整合实验室DAIRI）；Azienda Ospedaliera SS. Antonio e Biagio e Cesare Arrigo（安东尼和比阿乔和卡雷罗医院）
It is well noted that coordinate based MLPs benefit -- in terms of preserving high-frequency information -- through the encoding of coordinate positions as an array of Fourier features. Hitherto, the rationale for the effectiveness of these positional encodings has been solely studied through a Fourier lens. In this paper, we strive to broaden this understanding by showing that alternative non-Fourier embedding functions can indeed be used for positional encoding. Moreover, we show that their performance is entirely determined by a trade-off between the stable rank of the embedded matrix and t…
IIIT Sri City（印度信息技术学院斯里城）；UCLA（加州大学洛杉矶分校）；Wipro AI Labs（威普罗人工智能实验室）；AI Institute, University of South Carolina（南卡罗来纳大学人工智能研究所）
NLP applications for code-mixed (CM) or mix-lingual text have gained a significant momentum recently, the main reason being the prevalence of language mixing in social media communications in multi-lingual societies like India, Mexico, Europe, parts of USA etc. Word embeddings are basic build-ing blocks of any NLP system today, yet, word embedding for CM languages is an unexplored territory. The major bottleneck for CM word embeddings is switching points, where the language switches. These locations lack in contextually and statistical systems fail to model this phenomena due to high variance …
Machine learning has become a promising approach for molecular modeling. Positional quantities, such as interatomic distances and bond angles, play a crucial role in molecule physics. The existing works rely on careful manual design of their representation. To model the complex nonlinearity in predicting molecular properties in an more end-to-end approach, we propose to encode the positional quantities with a learnable embedding that is continuous and differentiable. A regularization technique is employed to encourage embedding smoothness along the physical dimension. We experiment with a vari…
Transformer models are permutation equivariant. To supply the order and type information of the input tokens, position and segment embeddings are usually added to the input. Recent works proposed variations of positional encodings with relative position encodings achieving better performance. Our analysis shows that the gain actually comes from moving positional information to attention layer from the input. Motivated by this, we introduce Decoupled Positional Attention for Transformers (DIET), a simple yet effective mechanism to encode position and segment information into the Transformer mod…
Deconstructing Positional Information: From Attention Logits to Training Biases
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Antai College of Economics and Management, Shanghai Jiao Tong University（上海交通大学安泰经济管理学院）
