忠实性作为信息流：评估与训练忠实的链式思维推理
Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning
AI总结：
通过信息流视角提出基于充分性、完整性和必要性的框架，结合熵、掩码KL和梯度诊断评估链式思维忠实性，并引入更新时干预（如注意力掩码、反向梯度掩码等）训练更忠实的推理模型。
摘要：
链式思维（CoT）推理仅在推理轨迹忠实反映产生最终答案的计算过程时，才有助于监控语言模型。然而，模型可能依赖绕过CoT的提示-答案捷径，使得可见的推理轨迹即使看似合理也具有误导性。我们通过结构化的信息流视角研究CoT忠实性：忠实推理应将答案相关信息通过从提示到CoT再到答案的中介路径路由，而非通过直接的提示-答案捷径。该视角产生了一个基于三个互补属性（充分性、完整性和必要性）的任务无关框架，我们使用基于熵的、掩码KL和基于梯度的诊断来实例化。我们表明，这些指标恢复了提示推理中外部判断的忠实性差异，并识别了基于KL的诊断中低熵失败模式，其中基于梯度的度量保持更稳定。基于此分析，我们引入了基于验证器的在线强化学习的更新时干预，包括注意力掩码、仅反向梯度掩码、CoT梯度以及提示表示的对抗扰动。在提示算术、可奖励黑客的代码修复以及未经提示训练但在错误提示注入下评估的DAPO-Math模型中，我们的干预将行为和结构指标转向更强的CoT中介。特别是，它们使捷径和奖励黑客行为在CoT中更加透明，并改善了任务无关的忠实性指标，同时在某些设置中也降低了对错误提示的敏感性。我们的结果表明，在训练期间控制信息流是通向更忠实和可监控的CoT推理的实用途径。代码见 https://github.com/safety-research/faithful-cot。
英文摘要：
Chain-of-thought (CoT) reasoning is useful for monitoring language models only when the reasoning trace faithfully reflects the computation that produces the final answer. However, models can rely on prompt-to-answer shortcuts that bypass the CoT, making the visible reasoning trace misleading even when it appears plausible. We study CoT faithfulness through a structural information-flow perspective: faithful reasoning should route answer-relevant information through the mediated path from prompt to CoT to answer, rather than through a direct prompt-to-answer shortcut. This perspective yields a task-agnostic framework based on three complementary properties, sufficiency, completeness, and necessity, which we instantiate with entropy-based, masked-KL, and gradient-based diagnostics. We show that these metrics recover externally judged faithfulness differences in hinted reasoning, and identify a low-entropy failure mode of KL-based diagnostics where gradient-based measures remain more stable. Building on this analysis, we introduce update-time interventions for verifier-based on-policy RL, including attention masking, backward-only gradient masking, CoT gradients, and adversarial perturbations of prompt representations. Across hinted arithmetic, reward-hackable code repair, and DAPO-Math models trained without hints but evaluated under wrong-hint injection, our interventions shift behavioral and structural indicators toward stronger CoT mediation. In particular, they make shortcut and reward-hacking behavior more transparent in the CoT and improve task-agnostic faithfulness metrics, while in some settings also reducing wrong-hint susceptibility. Our results suggest that controlling information flow during training is a practical route toward more faithful and monitorable CoT reasoning. Code is available at https://github.com/safety-research/faithful-cot.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
