Attention-based transformers have been remarkably successful at modeling generative processes across various domains and modalities. In this paper, we study the behavior of transformers on data drawn from \kth Markov processes, where the conditional distribution of the next symbol in a sequence depends on the previous $k$ symbols observed. We observe a surprising phenomenon empirically which contradicts previous findings: when trained for sufficiently long, a transformer with a fixed depth and $1$ head per layer is able to achieve low test loss on sequences drawn from \kth Markov sources, even as $k$ grows. Furthermore, this low test loss is achieved by the transformer's ability to represent and learn the in-context conditional empirical distribution. On the theoretical side, our main result is that a transformer with a single head and three layers can represent the in-context conditional empirical distribution for \kth Markov sources, concurring with our empirical observations. Along the way, we prove that \textit{attention-only} transformers with $O(\log_2(k))$ layers can represent the in-context conditional empirical distribution by composing induction heads to track the previous $k$ symbols in the sequence. These results provide more insight into our current understanding of the mechanisms by which transformers learn to capture context, by understanding their behavior on Markov sources.
What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains
Massachusetts Institute of Technology（麻省理工学院）；École Polytechnique Fédérale de Lausanne（瑞士联邦理工学院）；University of California, Berkeley（加州大学伯克利分校）；Princeton University（普林斯顿大学）
Depth-Width tradeoffs in Algorithmic Reasoning of Graph Tasks with Transformers
Courant Institute of Mathematical Sciences, New York University（纽约大学应用数学科学研究所）；Google Research（谷歌研究）；Meta AI；Bar-Ilan University（巴伊兰大学）；Department of Bio-Medical Engineering, Edmond J. Safra Center for Bioinformatics, Tel-Aviv University（生物医学工程系，埃德蒙·J·萨法中心，特拉维夫大学）；Tel Aviv University（特拉维夫大学）
Learning Policies for Markov Decision Processes from Data
Boston University（波士顿大学）；IIT-Bombay（印度理工学院孟买分校）；Zhejiang University（浙江大学）
我们认为学习的政策的马尔可夫决策过程与数据捕获的状态动作对遵循的政策的问题。我们假设该策略属于一类参数化策略，这些策略使用与状态-动作对相关联的特征来定义。这些特征是先验已知的，然而，只有它们中的未知子集可能是相关的。使用最佳拟合所观察的状态-动作样本的$\ell_1$ -正则化逻辑斯谛回归来恢复对应于所观察的目标策略的策略参数。我们建立的估计和原始政策（遗憾）的推广误差和遍历系数的基本马尔可夫链的平均回报之间的差异的界限。为此，我们结合样本复杂性理论和马尔可夫链平稳分布的灵敏度分析。我们的分析表明，要在$O(\sqrtε)$阶内实现后悔，使用$Ω(\log n \cdot poly(1/ε))$阶的训练样本量就足够了，其中$n$是特征的数量。我们证明了我们的方法的有效性合成机器人导航的例子。
Efficient Turing Machine Simulation with Transformers
Shenzhen International Center For Industrial And Applied Mathematics（深圳国际工业与应用数学中心）；Shenzhen Research Institute of Big Data（深圳大数据研究 institute）；CRRC Zhuzhou Institute（CRRC 长沙研究所）；Tengen Intelligence Institute（腾根智能研究院）
Deep Thinking by Markov Chain of Continuous Thoughts
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（认知智能国家重点实验室，中国科学技术大学）；Department of Statistics, University of Oxford（牛津大学统计系）；Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）；Hong Kong Institute of AI for Science, City University of Hong Kong（香港人工智能科学研究院，香港城市大学）
Differentiable Filtering for Learning Hidden Markov Models
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Coordinated Science Laboratory（协调科学实验室）；Department of Mechanical Science and Engineering（机械科学与工程系）
Bayesian Inverse Problems Meet Flow Matching: Efficient and Flexible Inference via Transformers
Skolkovo Institute of Science and Technology（斯克洛夫学院科学与技术研究所）；Sberbank（储蓄银行）；AI4S Center（AI4S中心）；Artificial Intelligence Research Institute（人工智能研究 institute）
BERT4Traj: Transformer Based Trajectory Reconstruction for Sparse Mobility Data
University of Georgia（佐治亚大学）；Department of Geography University of Georgia（佐治亚大学地理系）；College of Public Health University of Georgia（佐治亚大学公共卫生学院）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
