自回归训练如何导致不忠推理？合成实验研究
How Does Unfaithful Reasoning Emerge from Autoregressive Training? A Study of Synthetic Experiments
AI总结：
本研究通过合成实验探讨自回归训练如何导致不忠推理，发现训练噪声阈值影响推理的忠实性，模型在低噪声下能学习因果推理，高噪声下则出现跳步推理。
摘要：
链式推理（CoT）由大型语言模型（LLMs）生成时往往不忠：中间步骤可能在逻辑上不一致或未能反映导致最终答案的因果关系。尽管有大量实证观察，对CoT的理解仍不深入——什么是忠实的CoT推理，以及不忠性如何从自回归训练中产生。我们通过受控的合成实验研究这些问题，训练小型变压器在有噪声的数据上逐步解决模运算表达式，一项我们称为算术表达式推理的任务。我们发现，模型可以学习到忠实的推理，这种推理因果地遵循底层的算术规则，但只有在训练噪声低于临界阈值时才能实现，这一现象归因于简单性偏差。在更高的噪声水平下，训练动态会通过一种中间混合模式从忠实的逐步推理过渡到不忠的跳步推理，这种混合模式的特点是预测熵暂时增加。机制分析揭示，模型通过解决不一致的推理步骤来编码内部不确定性，这表明自回归训练中隐含的自我验证的出现。
英文摘要：
Chain-of-thought (CoT) reasoning generated by large language models (LLMs) is often unfaithful: intermediate steps can be logically inconsistent or fail to reflect the causal relationship leading to the final answer. Despite extensive empirical observations, a fundamental understanding of CoT is lacking--what constitutes faithful CoT reasoning, and how unfaithfulness emerges from autoregressive training. We study these questions using well-controlled synthetic experiments, training small transformers on noisy data to solve modular arithmetic expressions step by step, a task we term Arithmetic Expression Reasoning. We find that models can learn faithful reasoning that causally follows the underlying arithmetic rules, but only when the training noise is below a critical threshold, a phenomenon attributable to simplicity bias. At higher noise levels, training dynamics exhibit a transition from faithful stepwise reasoning to unfaithful skip-step reasoning via an intermediate mixed mode characterized by a transient increase in prediction entropy. Mechanistic analysis reveals that models learn to encode internal uncertainty by resolving inconsistent reasoning steps, which suggests the emergence of implicit self-verification from autoregressive training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
