Large language models based on the Transformer architecture have demonstrated impressive capabilities to learn in context. However, existing theoretical studies on how this phenomenon arises are limited to the dynamics of a single layer of attention trained on linear regression tasks. In this paper, we study the optimization of a Transformer consisting of a fully connected layer followed by a linear attention layer. The MLP acts as a common nonlinear representation or feature map, greatly enhancing the power of in-context learning. We prove in the mean-field and two-timescale limit that the infinite-dimensional loss landscape for the distribution of parameters, while highly nonconvex, becomes quite benign. We also analyze the second-order stability of mean-field dynamics and show that Wasserstein gradient flow almost always avoids saddle points. Furthermore, we establish novel methods for obtaining concrete improvement rates both away from and near critical points. This represents the first saddle point analysis of mean-field dynamics in general and the techniques are of independent interest.
Symmetry, Saddle Points, and Global Optimization Landscape of Nonconvex Matrix Factorization
University of Minnesota（明尼苏达大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Princeton University（普林斯顿大学）；Georgia Institute of Technology（佐治亚理工学院）
我们提出了一个一般性的理论来研究非凸\xl{optimization}的\xl{landscape}与潜在的对称结构\tz{for a class of machine learning problems (e.g., low-rank matrix factorization, phase retrieval, and deep linear neural networks)}。具体地，我们通过不变群\removed{for associated optimization problems, including low-rank matrix factorization, phase retrieval, and deep linear neural networks}的透镜刻画了Hessian矩阵\xl{of the objective function}的驻点位置和零空间。作为一个主要的激励例子，我们应用所提出的一般理论来表征全球\xl{landscape}的\xl{nonconvex optimization in}低秩矩阵分解问题。特别是，我们说明了如何旋转对称群产生无穷多个非孤立的严格鞍点和等价的全局极小的目标函数。通过显式识别所有静止点，我们将整个参数空间分为三个区域：（$\cR_1$）包含所有严格鞍点的邻域的区域，其中目标具有负曲率;（$\cR_2$）包含所有全局极…
Optimality and NP-Hardness of Transformers in Learning Markovian Dynamical Functions
Department of Computer Science, Rensselaer Polytechnic Institute（拉特格斯理工学院计算机科学系）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Mathematics of Computation, IBM Research（IBM研究院数学部）
Analysis of the Optimization Landscapes for Overcomplete Representation Learning
New York University（纽约大学）；UC Berkeley（加州大学伯克利分校）；ByteDance Inc.（字节跳动公司）；Chinese University of Hong Kong（香港中文大学）；Rutgers University（罗格斯大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Denver（丹佛大学）
Universal Dynamics of Warmup Stable Decay: understanding WSD beyond Transformers
MPI-IS Tübingen, Germany, ETH Zürich, Switzerland and Politecnico di Torino, Italy（马克斯·普朗克研究所（MPI-IS）图宾根，德国，瑞士苏黎世联邦理工学院（ETH Zürich）和意大利托里诺理工大学（Politecnico di Torino））；ETH Zürich, Switzerland（瑞士苏黎世联邦理工学院（ETH Zürich））；MPI-IS, ELLIS Institute Tübingen, Tübingen AI Center, Germany（马克斯·普朗克研究所（MPI-IS）图宾根，德国，图宾根人工智能中心（Tübingen AI Center））
Riemannian ascent--descent for nonconvex nonconcave minimax landscapes: convergence to basin saddle points and applications to distributionally robust optimization
To Transformers and Beyond: Large Language Models for the Genome
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；University Health Network（大学健康网络）；Lunenfeld-Tanenbaum Research Institute（卢南菲尔德-塔南鲍姆研究所）；University of California, San Francisco（加利福尼亚大学旧金山分校）
Bitformer: An efficient Transformer with bitwise operation-based attention for Big Data Analytics at low-cost low-precision devices
University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Advanced Research Institute, Chinese Academy of Sciences（中国科学院上海高等研究院）；Aston Business School, Aston University（阿斯顿大学阿斯顿商学院）
