Modern distributed networks, notably transformers, acquire a remarkable ability (termed `in-context learning') to adapt their computation to input statistics, such that a fixed network can be applied to data from a broad range of systems. Here, we provide a complete mechanistic characterization of this behavior in transformers trained on a finite set $S$ of discrete Markov chains. The transformer displays four algorithmic phases, characterized by whether the network memorizes and generalizes, and whether it uses 1-point or 2-point statistics. We show that the four phases are implemented by multi-layer subcircuits that exemplify two qualitatively distinct mechanisms for implementing context-adaptive computations. Minimal models isolate the key features of both motifs. Memorization and generalization phases are delineated by two boundaries that depend on data diversity, $K = |S|$. The first ($K_1^\ast$) is set by a kinetic competition between subcircuits and the second ($K_2^\ast$) is set by a representational bottleneck. A symmetry-constrained theory of a transformer's training dynamics explains the sharp transition from 1-point to 2-point generalization and identifies key features of the loss landscape that allow the network to generalize. Put together, we show that transformers develop distinct subcircuits to implement in-context learning and identify conditions that favor certain mechanisms over others.
The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Institute of Neuroinformatics, ETH Zürich and University of Zürich（神经信息学研究所，苏黎世联邦理工学院和苏黎世大学）；Centre for Artificial Intelligence, Zürich University of Applied Sciences（人工智能研究中心，应用科学大学）；ECLT European Centre for Living Technology, Venice, Italy（欧洲生命科技中心，威尼斯，意大利）；ETH AI Center, Zürich, Switzerland（苏黎世联邦理工学院AI中心）
Research Institute for Value-Added Information Generation（价值信息生成研究所）；JAMSTEC Center for Advanced Intelligence Project（JAMSTEC高级智能项目中心）；RIKEN（理化学研究所）；Graduate School of Information Science and Technology（信息科学与技术研究生院）；The University of Osaka（大阪大学）；Center for Advanced Intelligence Project（高级智能项目中心）
The Chess Transformer: Mastering Play using Generative Language Models
PeopleTec；Auburn University（奥本大学）
这项工作表明，自然语言Transformers可以支持更通用的战略建模，特别是对文本存档的游戏。除了学习自然语言技能之外，抽象的Transformer架构还可以在棋盘上生成有意义的移动。通过进一步的微调，Transformer通过在Portable Game Notation中对280万盘国际象棋进行训练来学习复杂的游戏玩法。经过30，000个训练步骤后，OpenAI的生成式预训练Transformer（GPT-2）优化了7.74亿个参数的权重。这款经过微调的国际象棋Transformer可以生成合理的策略，并显示可识别为经典开局的游戏阵型，例如英语或斯拉夫交换。最后，在现场比赛中，新模型展示了一个人对Transformer的界面，正确过滤非法移动，并提供了一种新的方法来挑战Transformer的象棋策略。我们预计未来的工作将建立在这个Transformer的承诺，特别是在其他战略游戏中，功能可以捕获简单但富有表现力的玩家注释的基础复杂的规则语法。
Physics Transformer: Tailoring Transformer for General PDE Prediction
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Mechanics and Engineering Science, Peking University（北京大学力学与工程科学学院）
Sparse Transformer Architectures via Regularized Wasserstein Proximal Operator with $L_1$ Prior
Department of Mathematics, University of California, Los Angeles, CA, USA（加州大学洛杉矶分校数学系）；Department of Mathematics, University of South Carolina, Columbia, SC, USA（南卡罗来纳大学数学系）
Understanding Generalization in Transformers: Error Bounds and Training Dynamics Under Benign and Harmful Overfitting
Zhongtai Securities Institute for Financial Studies, Shandong University（山东大学中泰证券金融研究院）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）
Transformers Provably Learn Directed Acyclic Graphs via Kernel-Guided Mutual Information
Department of Mathematics, National University of Singapore, Singapore（新加坡国立大学数学系）；Department of Statistics and Data Science, University of Pennsylvania, USA（宾夕法尼亚大学统计与数据科学系）；Department of Electrical and Computer Engineering, The Ohio State University, USA（俄亥俄州立大学电子与计算机工程系）
Optimality and NP-Hardness of Transformers in Learning Markovian Dynamical Functions
Department of Computer Science, Rensselaer Polytechnic Institute（拉特格斯理工学院计算机科学系）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Mathematics of Computation, IBM Research（IBM研究院数学部）
The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models
Skoltech（斯科尔科沃科技学院）；AIRI（人工智能研究所）；Lomonosov Moscow State University（罗蒙诺索夫莫斯科国立大学）；HSE University（高等经济大学）；SberAI（萨伯人工智能公司）；Samara National Research University（萨马拉国立研究大学）
Multi-Phase Spacecraft Trajectory Optimization via Transformer-Based Reinforcement Learning
Aeronautics & Astronautics, Massachusetts Institute of Technology（航空与宇航科学系，麻省理工学院）；Department of Computer Systems Engineering, Universidad Politécnica de Madrid（计算机系统工程系，马德里理工大学）
