反事实模拟训练用于思维链可信度
Counterfactual Simulation Training for Chain-of-Thought Faithfulness
AI总结：
本文提出反事实模拟训练方法，通过奖励使模拟器准确预测模型输出来提升思维链的可信度，并在不同设置中验证其有效性。
摘要：
检查思维链推理是理解LLM为何产生其输出最常用的方法之一。但众所周知的思维链可信度问题严重限制了从这种实践中获得的洞察。在本文中，我们介绍了一种称为反事实模拟训练（CST）的训练方法，旨在通过奖励使模拟器能够准确预测模型在反事实输入上的输出的思维链来提高思维链可信度。我们将在两种设置中应用CST：（1）基于提示的反事实的思维链监控，以检测模型是否依赖于虚假特征、奖励黑客或谄媚；（2）在通用模型基础上的反事实模拟，以鼓励模型在思维链中产生更可信、更具普遍性的推理。对高达235B参数的模型进行实验表明，CST可以显著提高基于提示的反事实的监控准确性（提升35个准确度点）以及在通用反事实上的可模拟性（提升2个点）。我们进一步表明：（1）CST优于提示基线；（2）用LLM重写不忠实的思维链比仅使用强化学习（RL）效率高5倍；（3）可信度的改进不适用于劝阻提示（而不是说服提示）；（4）更大的模型在出厂时并不更忠实，但它们从CST中受益更多。这些结果表明，CST可以普遍提高思维链的可信度，对思维链监控有令人鼓舞的应用前景。本文中实验的代码可在https://github.com/peterbhase/counterfactual-simulation-training获取。
英文摘要：
Inspecting Chain-of-Thought reasoning is among the most common means of understanding why an LLM produced its output. But well-known problems with CoT faithfulness severely limit what insights can be gained from this practice. In this paper, we introduce a training method called Counterfactual Simulation Training (CST), which aims to improve CoT faithfulness by rewarding CoTs that enable a simulator to accurately predict a model's outputs over counterfactual inputs. We apply CST in two settings: (1) CoT monitoring with cue-based counterfactuals, to detect when models rely on spurious features, reward hack, or are sycophantic, and (2) counterfactual simulation over generic model-based counterfactuals, to encourage models to produce more faithful, generalizable reasoning in the CoT. Experiments with models up to 235B parameters show that CST can substantially improve monitor accuracy on cue-based counterfactuals (by 35 accuracy points) as well as simulatability over generic counterfactuals (by 2 points). We further show that: (1) CST outperforms prompting baselines, (2) rewriting unfaithful CoTs with an LLM is 5x more efficient than RL alone, (3) faithfulness improvements do not generalize to dissuading cues (as opposed to persuading cues), and (4) larger models do not show more faithful CoT out of the box, but they do benefit more from CST. These results suggest that CST can improve CoT faithfulness in general, with promising applications for CoT monitoring. Code for experiments in this paper is available at https://github.com/peterbhase/counterfactual-simulation-training
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
