SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity
Broad Institute of MIT and Harvard（麻省理工学院与哈佛 broad 研究所）；Shanghai Jiao Tong University（上海交通大学）；University of Illinois, Chicago（芝加哥大学伊利诺伊分校）；Amazon Web Service（亚马逊网络服务）
Qitian Wu, Kai Yang, Hengrui Zhang, David Wipf, Junchi Yan
备注：
Extended version of NeurIPS2023 contribution arXiv:2306.10759
Learning representations on large graphs is a long-standing challenge due to the inter-dependence nature. Transformers recently have shown promising performance on small graphs thanks to its global attention for capturing all-pair interactions beyond observed structures. Existing approaches tend to inherit the spirit of Transformers in language and vision tasks, and embrace complicated architectures by stacking deep attention-based propagation layers. In this paper, we attempt to evaluate the necessity of adopting multi-layer attentions in Transformers on graphs, which considerably restricts the efficiency. Specifically, we analyze a generic hybrid propagation layer, comprised of all-pair attention and graph-based propagation, and show that multi-layer propagation can be reduced to one-layer propagation, with the same capability for representation learning. It suggests a new technical path for building powerful and efficient Transformers on graphs, particularly through simplifying model architectures without sacrificing expressiveness. As exemplified by this work, we propose a Simplified Single-layer Graph Transformers (SGFormer), whose main component is a single-layer global attention that scales linearly w.r.t. graph sizes and requires none of any approximation for accommodating all-pair interactions. Empirically, SGFormer successfully scales to the web-scale graph ogbn-papers100M, yielding orders-of-magnitude inference acceleration over peer Transformers on medium-sized graphs, and demonstrates competitiveness with limited labeled data.
We present a generalization of Transformers to any-order permutation invariant data (sets, graphs, and hypergraphs). We begin by observing that Transformers generalize DeepSets, or first-order (set-input) permutation invariant MLPs. Then, based on recently characterized higher-order invariant MLPs, we extend the concept of self-attention to higher orders and propose higher-order Transformers for order-$k$ data ($k=2$ for graphs and $k>2$ for hypergraphs). Unfortunately, higher-order Transformers turn out to have prohibitive complexity $\mathcal{O}(n^{2k})$ to the number of input nodes $n$. To …
Korea University（高丽大学）；NAVER CLOVA；NAVER AI LAB（NAVER人工智能实验室）
Transformer-based models have recently shown success in representation learning on graph-structured data beyond natural language processing and computer vision. However, the success is limited to small-scale graphs due to the drawbacks of full dot-product attention on graphs such as the quadratic complexity with respect to the number of nodes and message aggregation from enormous irrelevant nodes. To address these issues, we propose Deformable Graph Transformer (DGT) that performs sparse attention via dynamically sampled relevant nodes for efficiently handling large-scale graphs with a linear …
Locality-Sensitive Hashing-Based Efficient Point Transformer with Applications in High-Energy Physics
Georgia Institute of Technology（佐治亚理工学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Purdue University（普渡大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）
The University of Chicago（芝加哥大学）；Amazon Web Services（亚马逊网络服务）
Chain-of-thought (CoT) supervision can substantially improve transformer performance, yet the mechanisms by which models learn to follow and benefit from CoT remain poorly understood. We investigate these learning dynamics through the lens of grokking by pretraining transformers on symbolic reasoning tasks with tunable algorithmic complexity and controllable data composition to study their generalization. Models were trained under two settings: (i) producing only final answers, and (ii) emitting explicit CoT traces before answering. Our results show that while CoT generally improves task perfo…
School of Mathematical Sciences, Capital Normal University, 100048 Beijing, China（首都师范大学数学学院）；Advanced Institute of Information Technology, Peking University（北京大学信息科学技术高级研究院）；Wangxuan Institute of Computer Technology, Peking University, China, 100871（北京大学王先计算机技术研究所）；PKU-Wuhan Institute for Artificial Intelligence（北大-武汉人工智能研究所）；Xiangjiang Laboratory, Changsha 410000, China（湘江实验室）；Guanghua School of Management, Peking University, Beijing, China, 100871（北京大学光华管理学院）；School of Artificial Intelligence, Beihang University, 100191 Beijing, China（北航人工智能学院）
Institute of Informatics, LMU Munich（慕尼黑大学信息学研究所）；Corporate Technology, Siemens AG（西门子公司企业技术部）
Transformers have improved the state-of-the-art across numerous tasks in sequence modeling. Besides the quadratic computational and memory complexity w.r.t the sequence length, the self-attention mechanism only processes information at the same scale, i.e., all attention heads are in the same resolution, resulting in the limited power of the Transformer. To remedy this, we propose a novel and efficient structure named Adaptive Multi-Resolution Attention (AdaMRA for short), which scales linearly to sequence length in terms of time and space. Specifically, we leverage a multi-resolution multi-he…
Ister: Linear Transformer for Efficient Multivariate Time Series Forecasting
Hong Kong University of Science and Technology (Guangzhou)（香港理工大学（广州））；ShenZhen University（深圳大学）；Shenzhen Energy Group Co., Ltd.（深圳能源集团有限公司）；South China University of Technology（华南理工大学）
Politecnico di Milano（米兰理工大学）；International Telematic University Uninettuno（国际远程大学尤尼内图诺）
In this paper, we propose a two-steps approach to partition instances of the Conjunctive Normal Form (CNF) Syntactic Formula Isomorphism problem (CSFI) into groups of different complexity. First, we build a model, based on the Transformer architecture, that attempts to solve instances of the CSFI problem. Then, we leverage the errors of such model and train a second Transformer-based model to partition the problem instances into groups of different complexity, thus detecting the ones that can be solved without using too expensive resources. We evaluate the proposed approach on a pseudo-randoml…
Transformers based on the attention mechanism have achieved impressive success in various areas. However, the attention mechanism has a quadratic complexity, significantly impeding Transformers from dealing with numerous tokens and scaling up to bigger models. Previous methods mainly utilize the similarity decomposition and the associativity of matrix multiplication to devise linear-time attention mechanisms. They avoid degeneration of attention to a trivial distribution by reintroducing inductive biases such as the locality, thereby at the expense of model generality and expressiveness. In th…
End-to-end automatic speech recognition (ASR), unlike conventional ASR, does not have modules to learn the semantic representation from speech encoder. Moreover, the higher frame-rate of speech representation prevents the model to learn the semantic representation properly. Therefore, the models that are constructed by the lower frame-rate of speech encoder lead to better performance. For Transformer-based ASR, the lower frame-rate is not only important for learning better semantic representation but also for reducing the computational complexity due to the self-attention mechanism which has O…
Fudan University（复旦大学）；University of Surrey（萨里大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Vision transformers (ViTs) have pushed the state-of-the-art for various visual recognition tasks by patch-wise image tokenization followed by self-attention. However, the employment of self-attention modules results in a quadratic complexity in both computation and memory usage. Various attempts on approximating the self-attention computation with linear complexity have been made in Natural Language Processing. However, an in-depth analysis in this work shows that they are either theoretically flawed or empirically ineffective for visual recognition. We further identify that their limitations …
Circuit Complexity Bounds for RoPE-based Transformer Architecture
Middle Tennessee State University（中田纳西州立大学）；Stevens Institute of Technology（史蒂文斯理工学院）；The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；South China University of Technology（华南理工大学）；The Simons Institute for the Theory of Computing（西蒙斯理论计算研究所）；University of California, Berkeley（加利福尼亚大学伯克利分校）
Transformers have made progress in miscellaneous tasks, but suffer from quadratic computational and memory complexities. Recent works propose sparse Transformers with attention on sparse graphs to reduce complexity and remain strong performance. While effective, the crucial parts of how dense a graph needs to be to perform well are not fully explored. In this paper, we propose Normalized Information Payload (NIP), a graph scoring function measuring information transfer on graph, which provides an analysis tool for trade-offs between performance and complexity. Guided by this theoretical analys…
This paper analyzes three formal models of Transformer encoders that differ in the form of their self-attention mechanism: unique hard attention (UHAT); generalized unique hard attention (GUHAT), which generalizes UHAT; and averaging hard attention (AHAT). We show that UHAT and GUHAT Transformers, viewed as string acceptors, can only recognize formal languages in the complexity class AC$^0$, the class of languages recognizable by families of Boolean circuits of constant depth and polynomial size. This upper bound subsumes Hahn's (2020) results that GUHAT cannot recognize the DYCK languages or …
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Modern smart sensor-based energy management systems leverage non-intrusive load monitoring (NILM) to predict and optimize appliance load distribution in real-time. NILM, or energy disaggregation, refers to the decomposition of electricity usage conditioned on the aggregated power signals (i.e., smart sensor on the main channel). Based on real-time appliance power prediction using sensory technology, energy disaggregation has great potential to increase electricity efficiency and reduce energy expenditure. With the introduction of transformer models, NILM has achieved significant improvements i…
A Lightweight CNN Integrated Compact Convolutional Transformer for Multi-Scale Feature Learning and reducing computational complexity for breast cancer mammography image detection and classification
Australian National University（澳大利亚国立大学）；Google Brain（谷歌大脑）
The sparse transformer can reduce the computational complexity of the self-attention layers to $O(n)$, whilst still being a universal approximator of continuous sequence-to-sequence functions. However, this permutation variant operation is not appropriate for direct application to sets. In this paper, we proposed an $O(n)$ complexity sampled transformer that can process point set elements directly without any additional inductive bias. Our sampled transformer introduces random element sampling, which randomly splits point sets into subsets, followed by applying a shared Hamiltonian self-attent…
UniSymNet: A Unified Symbolic Network Guided by Transformer
School of Mathematical Sciences, East China Normal University（东华大学数学科学学院）；Institute of Artificial Intelligence, Beihang University（北航人工智能研究院）；Shanghai Zhangjiang Institute of Mathematics（上海张江数学研究所）；Institute of Applied Physics and Computational Mathematics（应用物理与计算数学研究所）；Academy for Advanced Interdisciplinary Studies, Northeast Normal University（东北师范大学先进跨学科研究学院）
Most research on pseudo relevance feedback (PRF) has been done in vector space and probabilistic retrieval models. This paper shows that Transformer-based rerankers can also benefit from the extra context that PRF provides. It presents PGT, a graph-based Transformer that sparsifies attention between graph nodes to enable PRF while avoiding the high computational complexity of most Transformer architectures. Experiments show that PGT improves upon non-PRF Transformer reranker, and it is at least as accurate as Transformer PRF models that use full attention, but with lower computational costs.
Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers
School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Institute for Advanced Algorithms Research, Shanghai Seres Information Technology Co., Ltd（上海赛尔斯信息技术有限公司高级算法研究院）
Autoregressive Transformers are strong language models but incur O(T) complexity during per-token generation due to the self-attention mechanism. Recent work proposes kernel-based methods to approximate causal self-attention by replacing it with recurrent formulations with various update rules and feature maps to achieve O(1) time and memory complexity. We explore these approaches and find that they are unnecessarily complex, and propose a simple alternative - decaying fast weights - that runs fast on GPU, outperforms prior methods, and retains 99% of attention's performance for GPT-2. We also…
Large transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, training and deploying these models can be prohibitively costly for long sequences, as the standard self-attention mechanism of the Transformer uses $O(n^2)$ time and space with respect to sequence length. In this paper, we demonstrate that the self-attention mechanism can be approximated by a low-rank matrix. We further exploit this finding to propose a new self-attention mechanism, which reduces the overall self-attention complexity from $…
University of California, Santa Barbara（加州大学圣巴巴拉分校）
Transformers are the mainstream of NLP applications and are becoming increasingly popular in other domains such as Computer Vision. Despite the improvements in model quality, the enormous computation costs make Transformers difficult at deployment, especially when the sequence length is large in emerging applications. Processing attention mechanism as the essential component of Transformer is the bottleneck of execution due to the quadratic complexity. Prior art explores sparse patterns in attention to support long sequence modeling, but those pieces of work are on static or fixed patterns. We…
Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time
The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Tsinghua University（清华大学）；The Simons Institute for the Theory of Computing at the University of California, Berkeley（加州大学伯克利分校西蒙斯理论计算研究所）；University of Pennsylvania（宾夕法尼亚大学）
Paris Saclay University（巴黎萨克雷大学）；Laboratory of Signal and System (L2S)（信号与系统实验室（L2S））
Following up on the linear transformer part of the article from Katharopoulos et al., that takes this idea from Shen et al., the trick that produces a linear complexity for the attention mechanism is re-used and extended to a second-order approximation of the softmax normalization.
Many real-world applications such as robotics provide hard constraints on power and compute that limit the viable model complexity of Reinforcement Learning (RL) agents. Similarly, in many distributed RL settings, acting is done on un-accelerated hardware such as CPUs, which likewise restricts model size to prevent intractable experiment run times. These "actor-latency" constrained settings present a major obstruction to the scaling up of model complexity that has recently been extremely successful in supervised learning. To be able to utilize large model capacity while still operating within …
Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decoding
The University of Chicago（芝加哥大学）；Stevens Institute of Technology（史蒂文斯理工学院）；The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；The Simons Institute for the Theory of Computing at UC Berkeley（加州大学伯克利分校西蒙斯理论计算研究所）
Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification
University of Copenhagen（哥本哈根大学）；Novo Nordisk A/S（诺华北欧制药有限公司）；Novo Nordisk Research Center Oxford Ltd.（诺华奥克斯研究中心）；Technical University of Denmark（丹麦技术大学）
Hierarchical Shift Mixing -- Beyond Dense Attention in Transformers
Dept. of Electrical Engineering, Linköping University, Linköping, Sweden（电子工程系，林霍普大学）；RISE Research Institutes of Sweden, Linköping, Sweden（瑞典RISE研究机构）
