Over the last years, state-tracking tasks, particularly permutation composition, have become a testbed to understand the limits of sequence models architectures like Transformers and RNNs (linear and non-linear). However, these are often sequence-to-sequence tasks: learning to map actions (permutations) to states, which is incompatible with the next-token prediction setting commonly used to train language models. We address this gap by converting permutation composition into code via REPL traces that interleave state-reveals through prints and variable transformations. We show that linear RNNs capable of state-tracking excel also in this setting, while Transformers still fail. Motivated by this representation, we investigate why tracking states in code is generally difficult: actions are not always fully observable. We frame this as tracking the state of a probabilistic finite-state automaton with deterministic state reveals and show that linear RNNs can be worse than non-linear RNNs at tracking states in this setup.
Fixed-Point RNNs: Interpolating from Diagonal to Dense
ELLIS Institute Tuebingen(图宾根ELLIS研究所);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);Department of Mathematics, Imperial College London(伦敦帝国学院数学系)
DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
University of Freiburg(弗赖堡大学);ELLIS Institute Tübingen(图宾根ELLIS研究所);Microsoft Research(微软研究院);Prior Labs(先驱实验室);CSML, Istituto Italiano di Tecnologia(CSML,意大利技术研究所);AI Centre, University College London(伦敦大学学院人工智能中心)
Implicit Language Models are RNNs: Balancing Parallelization and Expressivity
Chair of Highly-Parallel VLSI Systems and Neuro-Microelectronics , TUD Dresden University of Technology, Dresden, Germany(德累斯顿技术大学神经微电子学系);Microsoft Research, Cambridge, United Kingdom(微软研究院)
Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction
Interdisciplinary Center for Scientific Computing (IWR), Heidelberg University(海德堡大学跨学科科学计算中心);Faculty of Physics and Astronomy, Heidelberg University(海德堡大学物理与天文学院)
Improving Sequence Modeling Ability of Recurrent Neural Networks via Sememes
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Beijing University of Posts and Telecommunications(北京邮电大学)
Attention is All You Need? Good Embeddings with Statistics are enough:Large Scale Audio Understanding without Transformers/ Convolutions/ BERTs/ Mixers/ Attention/ RNNs or ....