Transformers were originally proposed as a sequence-to-sequence model for text but have become vital for a wide range of modalities, including images, audio, video, and undirected graphs. However, transformers for directed graphs are a surprisingly underexplored topic, despite their applicability to ubiquitous domains, including source code and logic circuits. In this work, we propose two direction- and structure-aware positional encodings for directed graphs: (1) the eigenvectors of the Magnetic Laplacian - a direction-aware generalization of the combinatorial Laplacian; (2) directional random walk encodings. Empirically, we show that the extra directionality information is useful in various downstream tasks, including correctness testing of sorting networks and source code understanding. Together with a data-flow-centric graph construction, our model outperforms the prior state of the art on the Open Graph Benchmark Code2 relatively by 14.7%.
CaTs and DAGs: Integrating Directed Acyclic Graphs with Transformers for Causally Constrained Predictions
Kivira Health(Kivira健康公司);The Sense Innovation and Research Center, CHUV, Lausanne, Switzerland(The Sense创新与研究中心,CHUV,洛桑,瑞士);Centre for Vision, Speech and Signal Processing, University of Surrey, U.K.(视觉、语音和信号处理中心,萨里大学,英国);Institute of Psychology, University of Lausanne, Switzerland(心理学研究所,洛桑大学,瑞士);EPFL, Switzerland(瑞士联邦理工学院)
Bayesian Inverse Problems Meet Flow Matching: Efficient and Flexible Inference via Transformers
Skolkovo Institute of Science and Technology(斯克洛夫学院科学与技术研究所);Sberbank(储蓄银行);AI4S Center(AI4S中心);Artificial Intelligence Research Institute(人工智能研究 institute)
Scalable Peptide Design via Memory-Efficient Equivariant Transformer
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);Department of Chemistry, Tsinghua University(清华大学化学系)
Tree Meets Transformer: A Hybrid Architecture for Scalable Power Allocation in Cell-Free Networks
Smart Networks and Services Joint Undertaking (SNS JU)(智能网络与服务联合体);Italian Ministry of Education and Research (MUR)(意大利教育与研究部);HORIZON-JU-SNS-2022 EU project TIMES(HORIZON-JU-SNS-2022 EU项目TIMES);European Union under the Italian National Recovery and Resilience Plan (NRRP) of NextGenerationEU(欧盟在意大利国家复苏与韧性计划(NRRP)下的NextGenerationEU)
Indian Institute of Information Technology Kottayam (IIITK)(印度信息技术学院科塔亚姆分校(IIITK));International Institute of Information Technology Hyderabad (IIITH)(印度信息技术学院海得拉巴分校(IIITH))
Transformers Meet In-Context Learning: A Universal Approximation Theory
Department of Statistics and Data Science, Chinese University of Hong Kong(统计与数据科学系,香港中文大学);the Wharton School, University of Pennsylvania(宾夕法尼亚大学沃顿商学院);University of Pennsylvania(宾夕法尼亚大学)
University of Electronic Science and Technology of China(电子科技大学);Clinical Hospital of Chengdu Brain Science Institute(成都脑科学医院);MOE-K Lab for NeuroInformation(神经信息MOE-K实验室);Int Institute of Brain-Apparatus Communication(脑机通讯国际研究所);School of Life Science and Technology(生命科学与技术学院)
Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
Department of Computer Science and Engineering, Korea University, Seoul, South Korea(计算机科学与工程系,韩国大学,首尔,韩国);School of Software, Soongsil University, Seoul, South Korea(软件学院,顺天大学,首尔,韩国)
Transformers Provably Learn Directed Acyclic Graphs via Kernel-Guided Mutual Information
Department of Mathematics, National University of Singapore, Singapore(新加坡国立大学数学系);Department of Statistics and Data Science, University of Pennsylvania, USA(宾夕法尼亚大学统计与数据科学系);Department of Electrical and Computer Engineering, The Ohio State University, USA(俄亥俄州立大学电子与计算机工程系)
LNML-based Causal Discovery with Latent Confounders: Theory and Experiments
Graduate School of Information Science and Technology, The University of Tokyo(东京大学信息科学与技术研究生院);Graduate School of Arts and Sciences, The University of Tokyo(东京大学文理研究生院)