We introduce directional routing, a lightweight mechanism that gives each transformer attention head learned suppression directions controlled by a shared router, at 3.9% parameter cost. We train a 433M-parameter model alongside an identical baseline in a single run, then trace the resulting circuits through mechanistic interpretability.
Routing becomes the model's dominant computational pathway. Disabling it collapses factual recall to near-zero probability across all 8 test prompts and drops induction accuracy from 93.4% to 0.0%. Knocking out individual attention heads has negligible effect: the primary mover head's removal actually increases target probability, and induction heads retain 98.6% accuracy without their strongest member. The coordination mechanism is irreplaceable; the components it coordinates are not. The model also self-organizes, without explicit pressure, into two regimes: domain-adaptive routing in early layers and fixed syntactic pruning in late layers, where the least-varying layer is the most critical (+42.6 PPL when disabled). Routing reduces perplexity 31-56% relative to the baseline, though downstream multiple-choice benchmarks do not yet reflect these gains.
Learning to Iteratively Solve Routing Problems with Dual-Aspect Collaborative Transformer
National University of Singapore(新加坡国立大学);University of Electronic Science and Technology of China(电子科技大学);Institute for Infocomm Research, A*STAR(新加坡科技研究局资讯通信研究院);The Hong Kong University of Science and Technology(香港科技大学);Singapore Institute of Manufacturing Technology, A*STAR(新加坡科技研究局新加坡制造技术研究院);Shandong University(山东大学)
The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Institute of Neuroinformatics, ETH Zürich and University of Zürich(神经信息学研究所,苏黎世联邦理工学院和苏黎世大学);Centre for Artificial Intelligence, Zürich University of Applied Sciences(人工智能研究中心,应用科学大学);ECLT European Centre for Living Technology, Venice, Italy(欧洲生命科技中心,威尼斯,意大利);ETH AI Center, Zürich, Switzerland(苏黎世联邦理工学院AI中心)
Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers
University of California San Diego(加州大学圣迭戈分校);University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Fermi National Accelerator Laboratory(费米国家加速器实验室);University of California Davis(加州大学戴维斯分校)
Efficient Content-Based Sparse Attention with Routing Transformers
Google Research(谷歌研究院)
自注意最近已被广泛的序列建模问题。尽管它的有效性,自我注意遭受二次计算和存储器的要求相对于序列长度。降低这种复杂性的成功方法集中在关注本地滑动窗口或独立于内容的一小部分位置。我们的工作提出了学习动态稀疏注意模式,避免分配计算和内存参加内容无关的查询的兴趣。这项工作建立在两条线的研究:它结合了先前的工作基于内容的稀疏注意力的建模灵活性,从本地的,时间稀疏注意力的方法的效率收益。我们的模型,路由Transformer,赋予自我注意力的稀疏路由模块的基础上在线k-均值,同时降低整体的复杂性注意$O\left(n^{1.5}d\right)$从$O\left(n^2d\right)$的序列长度$n$和隐藏的维度$d$。我们表明,我们的模型在Wikitext-103上的语言建模(15.8 vs 18.3困惑度)以及ImageNet-64上的图像生成(3.43 vs 3.44 bits/dim)方面优于可比较的稀疏注意模型,同时使用较少的自我注意层。此外,我们在新发布的PG-19数据集上设置了一个新的最新技术水平,使用在长度为8192的序列上训练的22层Routing Transformer模型获得了33.2的测试困惑度。
Adversarial Generative Flow Network for Solving Vehicle Routing Problems
Singapore Management University(新加坡管理大学);Singapore Institute of Manufacturing Technology (SIMTech), Agency of Science, Technology and Research (A*STAR)(新加坡制造技术研究院(新加坡科技研究局下属机构))
Geometric Capacity of Transformers: A Tropical Geometry Perspective
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院)
Adaptive Routing for Efficient Diffusion Transformer-Based PNI Prediction
Seoul National University(首尔国立大学);Kyung Hee University(庆熙大学);OUTTA;Chung-Ang University(Chung-Ang 大学);Seoul National University School of Medicine(首尔国立大学医学院);Samsung Changwon Hospital(三星昌原医院);Samsung Medical Center(三星医疗中心);NVIDIA AI Technology Center(NVIDIA AI 技术中心)
EFormer: An Effective Edge-based Transformer for Vehicle Routing Problems
School of Computer Science and Technology, Dalian University of Technology (DUT)(大连理工大学计算机科学与技术学院);School of Computing and Information Systems, Singapore Management University(新加坡管理学院);Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology(埃因霍温理工大学工业工程与创新科学系);Key Laboratory of Social Computing and Cognitive Intelligence (DUT), Ministry of Education, China(社会计算与认知智能重点实验室(DUT))
Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Fudan University, Shanghai, China(复旦大学);University of Bath, Bath, United Kingdom(巴斯大学);Oxford Suzhou Centre for Advanced Research, Suzhou, China(牛津苏滁研究中心);Shanghai Innovation Institute, Shanghai, China(上海创新研究院);Department of Computer Science, University of Colorado Boulder, Colorado, USA(计算机科学系,科罗拉多大学博尔德分校);Department of Electrical Engineering and Computer Science, University of Michigan(电气工程与计算机科学系,密歇根大学)
DTA-GT: Direction- and Topology-Aware Graph Transformer for Neural Network Representation Learning
Hunan University(湖南大学);Hunan University of Science and Technology(湖南科技大学);Chinese Academy of Sciences(中国科学院);The Hong Kong Polytechnic University(香港理工大学)
Bi-Directional Transformers vs. word2vec: Discovering Vulnerabilities in Lifted Compiled Code
The Beacom College of Computer and Cyber Sciences, Dakota State University(达科他州立大学比科姆计算机与网络科学学院);College of Applied Science & Technology, University of Arizona(亚利桑那大学应用科学与技术学院);College of Computing, Georgia Institute of Technology(佐治亚理工学院计算学院)