Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens. This representation is then exploited by the attention function, which learns dependencies between tokens and is key to the success of Transformers. However, the iterative application of attention across layers induces complex dynamics that remain to be fully understood. To analyze these dynamics, we identify each input sequence with a probability measure and model its evolution as a Vlasov equation called Transformer PDE, whose velocity field is non-linear in the probability measure. Our first set of contributions focuses on compactly supported initial data. We show the Transformer PDE is well-posed and is the mean-field limit of an interacting particle system, thus generalizing and extending previous analysis to several variants of self-attention: multi-head attention, L2 attention, Sinkhorn attention, Sigmoid attention, and masked attention--leveraging a conditional Wasserstein framework. In a second set of contributions, we are the first to study non-compactly supported initial conditions, by focusing on Gaussian initial data. Again for different types of attention, we show that the Transformer PDE preserves the space of Gaussian measures, which allows us to analyze the Gaussian case theoretically and numerically to identify typical behaviors. This Gaussian analysis captures the evolution of data anisotropy through a deep Transformer. In particular, we highlight a clustering phenomenon that parallels previous results in the non-normalized discrete case.
Department of Mathematics, Massachusetts Institute of Technology, 77 Massachusetts Ave, 02139 Cambridge MA, USA；CNRS \& Université Paris-Saclay Laboratoire de mathématiques d'Orsay, 307 rue Michel Magat, Bâtiment 307, 91400 Orsay, France；Department of EECS, Massachusetts Institute of Technology, 77 Massachusetts Ave, 02139 Cambridge MA, USA
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
Center for Data Science（数据科学中心）；Peking University（北京大学）；Institute for Functional Intelligent Materials（功能智能材料研究所）；National University of Singapore（新加坡国立大学）；Department of Mathematics（数学系）；Microsoft Research Asia（微软亚洲研究院）
Dual-perspective Cross Contrastive Learning in Graph Transformers
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；School of Computing, Macquarie University（麦考瑞大学计算学院）
Geometric Capacity of Transformers: A Tropical Geometry Perspective
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）
Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；The Simons Institute for the Theory of Computing（西蒙斯理论计算研究所）；UC Berkeley（加州大学伯克利分校）；Sun Yat-sen University（中山大学）
Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Hubei Key Laboratory of Computational Science, Wuhan University（湖北省计算科学重点实验室）；Department of Mathematics, The Hong Kong University of Science and Technology（香港科技大学数学系）
A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective
The University of Hong Kong（香港大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Future Network of Intelligence Institute, CUHK-Shenzhen（香港中文大学（深圳）未来智联网络研究院）；ShanghaiTech University（上海科技大学）
UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer
Northeastern University（东北大学）；Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University（东北大学医学影像智能计算教育部重点实验室）；Amii, University of Alberta（阿尔伯塔大学阿尔伯塔机器智能研究所）
Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Models
Department of Mathematics at the University of Maryland, College Park（马里兰大学College Park数学系）；Human Language Technology Center of Excellence, Johns Hopkins University（约翰霍普金斯大学人机语言技术中心）；Center for Imaging Science (CIS), the Institute for Computational Medicine (ICM), and the Mathematical Institute for Data Science (MINDS), Johns Hopkins University（约翰霍普金斯大学影像科学中心（CIS）、计算医学研究所（ICM）和数据科学数学研究所（MINDS））；Department of Applied Mathematics and Statistics (AMS), the Center for Imaging Science (CIS), and the Mathematical Institute for Data Science (MINDS), Johns Hopkins University（约翰霍普金斯大学应用数学与统计学系（AMS）、影像科学中心（CIS）和数据科学数学研究所（MINDS））
Efficient Large-Scale Traffic Forecasting with Transformers: A Spatial Data Management Perspective
University of Electronic Science and Technology of China（电子科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Auburn University（奥本大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；National University of Singapore（新加坡国立大学）；Peking University（北京大学）；Yangtze Delta Region Institute (Quzhou)（长三角研究院（衢州））
