Transformer 内部结合思维链的有限状态自动机：状态追踪的机制研究
Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Tracking
AI总结：
本文通过机制研究证明 Transformer+CoT 在状态追踪任务中隐式学习到有限状态自动机，并发现后期层 MLP 神经元是关键电路，且在跳过步骤、数据噪声和长度泛化等挑战下仍保持鲁棒性。
摘要：
思维链（CoT）显著提升了大语言模型（LLM）在广泛任务上的表现，先前研究表明 CoT 在理论上可以增强表达能力。然而，对于 Transformer+CoT 能够学习到的算法，目前仍缺乏机制层面的理解。我们的主要贡献如下：（1）我们评估了 Transformer+CoT 及其变体的状态追踪能力，证实了 CoT 的有效性。（2）接着，我们识别出负责追踪世界状态的电路（模型组件的一个子集），表明后期层的 MLP 神经元发挥了关键作用。我们提出压缩性和区分性两个指标，并证明每个状态对应的神经元集合达到接近 100% 的准确率，为模型内部隐式嵌入有限状态自动机（FSA）提供了证据。（3）此外，我们探索了三种具有挑战性的设置：跳过中间步骤、引入数据噪声以及测试长度泛化。结果表明 Transformer+CoT 学习到了鲁棒的算法（FSA），凸显了其在挑战性场景下的韧性。代码可在 https://github.com/IvanChangPKU/FSA 获取。
英文摘要：
Chain-of-thought (CoT) significantly enhances the performance of large language models (LLMs) across a wide range of tasks, and prior research shows that CoT can theoretically increase expressiveness. However, there is limited mechanistic understanding of the algorithms that Transformer+CoT can learn. Our key contributions are: (1) We evaluate the state tracking capabilities of Transformer+CoT and its variants, confirming the effectiveness of CoT. (2) Next, we identify the circuit (a subset of model components, responsible for tracking the world state), indicating that late-layer MLP neurons play a key role. We propose two metrics, compression and distinction, and show that the neuron sets for each state achieve nearly 100% accuracy, providing evidence of an implicit finite state automaton (FSA) embedded within the model. (3) Additionally, we explore three challenging settings: skipping intermediate steps, introducing data noises, and testing length generalization. Our results demonstrate that Transformer+CoT learns robust algorithms (FSAs), highlighting its resilience in challenging scenarios. Our code is available at https://github.com/IvanChangPKU/FSA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
