Attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages. To deepen our understanding of their sequential modeling capabilities, there is a growing interest in using Markov input processes to study them. A key finding is that when trained on first-order Markov chains, transformers with two or more layers consistently develop an induction head mechanism to estimate the in-context bigram conditional distribution. In contrast, single-layer transformers, unable to form an induction head, directly learn the Markov kernel but often face a surprising challenge: they become trapped in local minima representing the unigram distribution, whereas deeper models reliably converge to the ground-truth bigram. While single-layer transformers can theoretically model first-order Markov chains, their empirical failure to learn this simple kernel in practice remains a curious phenomenon. To explain this contrasting behavior of single-layer models, in this paper we introduce a new framework for a principled analysis of transformers via Markov chains. Leveraging our framework, we theoretically characterize the loss landscape of single-layer transformers and show the existence of global minima (bigram) and bad local minima (unigram) contingent on data properties and model architecture. We precisely delineate the regimes under which these local optima occur. Backed by experiments, we demonstrate that our theoretical findings are in congruence with the empirical results. Finally, we outline several open problems in this arena. Code is available at https://github.com/Bond1995/Markov .
PAINET: A Principled Efficient Transformer for 3D Dynamics Modeling
Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University（上海交通大学巴黎精英理工学院）；Department of Physics, Harvard University（哈佛大学物理系）；Harvard-MIT Center for Ultracold Atoms（哈佛-麻省理工冷原子中心）；Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard（MIT和哈佛大学Broad研究所埃里克和wendy Schmidt中心）
What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains
Massachusetts Institute of Technology（麻省理工学院）；École Polytechnique Fédérale de Lausanne（瑞士联邦理工学院）；University of California, Berkeley（加州大学伯克利分校）；Princeton University（普林斯顿大学）
Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signals
Department of Mathematics, University of Oxford（牛津大学数学系）；Oxford Man Institute for Quantitative Finance, University of Oxford（牛津大学牛津曼量化金融研究所）；Department of Mathematics, McMaster University（麦克马斯特大学数学系）；Vector Institute（向量研究所）
Likelihood-guided Regularization in Attention Based Models
Department of Statistics, Virginia Polytechnic Institute and State University（统计学系，弗吉尼亚理工大学和州立大学）；Department of Statistics, Virginia Tech.（统计学系，弗吉尼亚理工大学）
QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；Institute of Artificial Intelligence（人工智能研究院）；Robotics, Xi'an Jiaotong University（机器人技术，西安交通大学）；Computer Science, University College London（计算机科学，伦敦大学学院）；School of Engineering, Westlake University（工程学院，西湖大学）
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
Center for Data Science（数据科学中心）；Peking University（北京大学）；Institute for Functional Intelligent Materials（功能智能材料研究所）；National University of Singapore（新加坡国立大学）；Department of Mathematics（数学系）；Microsoft Research Asia（微软亚洲研究院）
Deep Thinking by Markov Chain of Continuous Thoughts
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（认知智能国家重点实验室，中国科学技术大学）；Department of Statistics, University of Oxford（牛津大学统计系）；Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）；Hong Kong Institute of AI for Science, City University of Hong Kong（香港人工智能科学研究院，香港城市大学）
A unified framework for establishing the universal approximation of transformer-type architectures
Department of Mathematics National University of Singapore（新加坡国立大学数学系）；School of Mathematical Sciences Peking University（北京大学数学科学学院）；Department of Mathematics Institute for Functional Intelligent Materials National University of Singapore（新加坡国立大学功能智能材料研究所数学系）
