Despite the remarkable practical success of transformer-based language models, recent work has raised concerns about their ability to perform state tracking. In particular, a growing body of literature has shown this limitation primarily through failures in out-of-distribution (OOD) generalization, such as length extrapolation. In this work, we shift attention to the in-distribution implications of these limitations. We conduct a large-scale experimental study of the data efficiency of transformers and recurrent neural networks (RNNs) across multiple supervision regimes. We find that the amount of training data required by transformers grows much more rapidly with state-space size and sequence length than for RNNs. Furthermore, we analyze the extent to which learned state-tracking mechanisms are shared across different sequence lengths. We show that transformers exhibit negligible or even detrimental weight sharing across lengths, indicating that they learn length-specific solutions in isolation. In contrast, recurrent models exhibit effective amortized learning by sharing weights across lengths, allowing data from one sequence length to improve performance on others. Together, these results demonstrate that state tracking remains a fundamental challenge for transformers, even when training and evaluation distributions match.
From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers
The University of Tokyo(东京大学);RIKEN AIP(理化学研究所AIP);Flatiron Institute(Flatiron研究所);CNRS(法国国家科学研究中心);Université Côte d’Azur(蔚蓝海岸大学);New York University(纽约大学)
Path-dependent program induction under resource constraints explains human sequence learning
University of Tübingen(蒂宾根大学);Center for Cognitive Science, Technical University Darmstadt(达姆施塔特工业大学认知科学中心);Department of Computational Neuroscience, Max Planck Institute for Biological Cybernetics(马克斯·普朗克生物控制论研究所计算神经科学系)
Differentiable Rule Induction from Raw Sequence Inputs
Institute of High Performance Computing, Agency for Science, Technology and Research(高性能计算研究所,科技研究局);National Institute of Informatics(国家信息研究所);Key Laboratory of High Confidence Software Technologies, School of Computer Science Peking University(高可信软件技术重点实验室,北京大学计算机学院)
What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains
Massachusetts Institute of Technology(麻省理工学院);École Polytechnique Fédérale de Lausanne(瑞士联邦理工学院);University of California, Berkeley(加州大学伯克利分校);Princeton University(普林斯顿大学)