arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.14814cs.LGcs.CL
收藏

利用线性RNN从代码中学习状态追踪

Learning State-Tracking from Code Using Linear RNNs

Julien Siems, Riccardo Grazzi, Korbinian Pöppel, Kirill Kalinin, Hitesh Ballani, Babak Rahmani

AI总结:

通过将置换组合任务转换为带状态揭示的代码,研究线性RNN在状态追踪中的表现,发现其优于Transformer,但在部分可观测状态下不如非线性RNN。

摘要:

在过去的几年中,状态追踪任务,特别是置换组合,已成为理解序列模型架构(如Transformer和RNN(线性和非线性))局限性的测试平台。然而,这些通常是序列到序列的任务:学习将动作(置换)映射到状态,这与常用于训练语言模型的下一个词元预测设置不兼容。我们通过将置换组合转换为代码来弥补这一差距,这些代码通过打印和变量变换的REPL轨迹交错显示状态。我们表明,能够进行状态追踪的线性RNN在此设置中也表现出色,而Transformer仍然失败。受这种表示的启发,我们研究了为什么在代码中追踪状态通常很困难:动作并不总是完全可观测的。我们将此框架化为追踪具有确定性状态揭示的概率有限状态自动机的状态,并表明线性RNN在此设置中追踪状态的能力可能不如非线性RNN。

英文摘要:

Over the last years, state-tracking tasks, particularly permutation composition, have become a testbed to understand the limits of sequence models architectures like Transformers and RNNs (linear and non-linear). However, these are often sequence-to-sequence tasks: learning to map actions (permutations) to states, which is incompatible with the next-token prediction setting commonly used to train language models. We address this gap by converting permutation composition into code via REPL traces that interleave state-reveals through prints and variable transformations. We show that linear RNNs capable of state-tracking excel also in this setting, while Transformers still fail. Motivated by this representation, we investigate why tracking states in code is generally difficult: actions are not always fully observable. We frame this as tracking the state of a probabilistic finite-state automaton with deterministic state reveals and show that linear RNNs can be worse than non-linear RNNs at tracking states in this setup.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑