在生成链式推理前解码答案：来自预CoT探测器和激活引导的证据
Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers
AI总结：
研究发现，模型在生成链式推理前已确定答案，通过激活引导实验揭示了预CoT过程中的因果关系及潜在错误模式。
摘要：
随着链式推理（CoT）成为在大型语言模型（LLMs）中扩展推理能力的核心，它也作为一种有前景的可解释性工具出现，表明通过口头化推理来理解模型决策的机会。然而，CoT在可解释性中的效用取决于其忠实性——即模型所陈述的推理是否反映底层的决策过程。我们提供了机制证据，表明经过指令微调的模型往往在生成CoT之前确定其答案。在训练线性探测器时，我们对生成CoT前最后一个标记的残差流激活进行训练，可以在大多数任务上以0.9的AUC预测模型的最终答案。我们发现这些方向不仅具有预测性，而且具有因果性：将激活沿探测器方向引导会在超过50%的情况下翻转模型答案，显著超过正交基线。当引导导致错误答案时，我们观察到两种不同的失败模式：非蕴含（陈述正确的前提但得出不支持的结论）和虚构（编造虚假的前提）。虽然事后推理可能在模型有正确预CoT信念时具有工具性用途，但这些失败模式表明，当从错误信念进行推理时可能会产生不可接受的行为。
英文摘要：
As chain of thought (CoT) has become central to scaling reasoning capabilities in large language models (LLMs), it has also emerged as a promising tool for interpretability, suggesting the opportunity to understand model decisions through verbalized reasoning. However, the utility of CoT toward interpretability depends upon its faithfulness---whether the model's stated reasoning reflects the underlying decision process. We provide mechanistic evidence that instruction-tuned models often determine their answer before generating CoT. Training linear probes on residual stream activations at the last token before CoT, we can predict the model's final answer with >0.9 AUC on most tasks. We find that these directions are not only predictive, but also causal: steering activations along the probe direction often flips model answers, with flip rates substantially exceeding norm-matched orthogonal baselines across most model-dataset pairs. When steering induces incorrect answers, we observe two distinct failure modes: confabulation (fabricating false premises) and non-entailment (stating correct premises but drawing unsupported conclusions). While post-hoc reasoning may be instrumentally useful when the model has a correct pre-CoT belief, these failure modes suggest it can result in undesirable behaviors when reasoning from a false belief.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
