Empirical studies have identified a range of learnability biases and limitations of transformers, such as a persistent difficulty in learning to compute simple formal languages such as PARITY, and a bias towards low-degree functions. However, theoretical understanding remains limited, with existing expressiveness theory either overpredicting or underpredicting realistic learning abilities. We prove that, under the transformer architecture, the loss landscape is constrained by the input-space sensitivity: Transformers whose output is sensitive to many parts of the input string inhabit isolated points in parameter space, leading to a low-sensitivity bias in generalization. We show theoretically and empirically that this theory unifies a broad array of empirical observations about the learning abilities and biases of transformers, such as their generalization bias towards low sensitivity and low degree, and difficulty in length generalization for PARITY. This shows that understanding transformers' inductive biases requires studying not just their in-principle expressivity, but also their loss landscape.
Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention
Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；State Key laboratory of Space Network and Communications（空间网络与通信国家重点实验室）
Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility
Center for Applied Mathematics（应用数学中心）；Cornell University（康奈尔大学）；Amazon Web Services（亚马逊网络服务）；Amazon Selling Partner Services（亚马逊销售合作伙伴服务）；Amazon Supply Chain Optimization Technologies（亚马逊供应链优化技术）
On Layer Normalization in the Transformer Architecture
Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；Beijing Institute of Big Data Research（北京大数据研究院）；Microsoft Research（微软研究院）；Nankai University（南开大学）
Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；The Simons Institute for the Theory of Computing（西蒙斯理论计算研究所）；UC Berkeley（加州大学伯克利分校）；Sun Yat-sen University（中山大学）
Why "classic" Transformers are shallow and how to make them go deep
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
Locality-Sensitive Hashing-Based Efficient Point Transformer with Applications in High-Energy Physics
Georgia Institute of Technology（佐治亚理工学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Purdue University（普渡大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）
Improved context-sensitive transformer model for inland vessel trajectory prediction
Institute of Ship Technology, Ocean Engineering and Transport Systems (ISMT), University of Duisburg-Essen（杜伊斯堡-埃森大学船舶技术、海洋工程与运输系统研究所（ISMT））；German Federal Waterways Engineering and Research Institute（德国联邦航道工程与研究院）；University of Duisburg-Essen（杜伊斯堡-埃森大学）
Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis
Rensselaer Polytechnic Institute（伦斯勒理工学院）；New Jersey Institute of Technology（新泽西理工学院）；Michigan State University（密歇根州立大学）；IBM Thomas J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Locality-Sensitive Hashing-Based Efficient Point Transformer for Charged Particle Reconstruction
Georgia Institute of Technology（佐治亚理工学院）；Purdue University（普渡大学）；University of Washington（华盛顿大学）；Princeton University（普林斯顿大学）；University of California San Diego（加州大学圣地亚哥分校）
FairGP: A Scalable and Fair Graph Transformer Using Graph Partitioning
Dalian University of Technology（大连理工大学）；University of South Australia（南澳大学）；The University of New South Wales（新南威尔士大学）；The University of Melbourne（墨尔本大学）；RMIT University（皇家墨尔本理工大学）
Do Transformers Have the Ability for Periodicity Generalization?
School of Computer Science, Peking University（北京大学计算机科学学院）；College of AI, Tsinghua University（清华大学人工智能学院）；School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）科学与工程学院）
Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal
Indian Institute of Information Technology Allahabad, India（阿利哈巴德信息技术学院，印度）；National Institute of Electronics（电子研究所）；Indian Institute of Technology Patna, India（帕纳印度理工学院，印度）；Indian Institute of Information Technology Kalyani, India（卡里尼信息技术学院，印度）
Distributional Drift Detection in Medical Imaging with Sketching and Fine-Tuned Transformer
Department of Neurology, University of Miami, FL USA（迈阿密大学神经病学系）；Department of Computer Science & IDSC, University of Miami, FL USA（迈阿密大学计算机科学与IDSC系）；Georgetown University Medical Center, Washington D.C. USA（乔治城大学医学中心）
