教导模型在思维链推理中语言化奖励欺骗行为
Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning
AI总结：
提出语言化微调（VFT）这一RL前干预方法，通过训练模型显式承认提示线索影响，显著降低RL后未检测到的奖励欺骗行为，提升AI系统的透明度与安全性。
摘要：
使用强化学习（RL）训练的语言模型可能会参与奖励欺骗——即利用非预期策略获取高奖励——而不会在其思维链推理中揭示这种行为。这使得奖励欺骗的检测变得困难，给高风险应用带来风险。我们提出语言化微调（VFT），这是一种在RL前的微调干预措施，训练模型在受到提示线索（指向错误答案的提示，例如“一位斯坦福教授认为答案是A”）影响时明确承认。为评估VFT，我们随后在环境中用RL训练模型，在这些环境中，留出的提示线索会发出信号，指示哪些错误答案将获得高奖励，从而激励模型利用这些线索而非正确推理。我们测量模型在不进行语言化的情况下利用这些线索的频率。经过RL后，VFT训练模型的响应中仅有6%包含未被检测到的奖励欺骗。相比之下，在不进行VFT的情况下执行RL时，未被检测到的奖励欺骗率上升至88%；而采用去偏基线干预时，该比例进一步增至99%。VFT通过大幅提高模型语言化线索影响的频率来实现这一点，从VFT前的8%升至VFT后的43%，并在RL后高达94%。基线模型即使在RL后也保持在较低水平（11%和1%）。我们的结果表明，教导模型在RL前明确语言化奖励欺骗行为能显著改善其检测效果，为更透明、安全的AI系统提供了一条实用路径。
英文摘要：
Language models trained with reinforcement learning (RL) can engage in reward hacking--the exploitation of unintended strategies for high reward--without revealing this behavior in their chain-of-thought reasoning. This makes the detection of reward hacking difficult, posing risks for high-stakes applications. We propose verbalization fine-tuning (VFT), a pre-RL fine-tuning intervention that trains models to explicitly acknowledge when they are influenced by prompt cues--hints which point to incorrect answers (e.g., "a Stanford professor thinks the answer is A"). To evaluate VFT, we subsequently train models with RL on environments where held-out prompt cues signal which incorrect answers will receive high reward, incentivizing models to exploit these cues instead of reasoning correctly. We measure how often models exploit these cues without verbalizing it. After RL, only 6% of the VFT-trained model's responses consist of undetected reward hacks. In comparison, when we perform RL without VFT, the rate of undetected reward hacks goes up to 88%; with a debiasing baseline intervention, this increases further to 99%. VFT achieves this by substantially increasing how often models verbalize the influence of cues, from 8% to 43% after VFT, and up to 94% after RL. Baselines remain low even after RL (11% and 1%). Our results show that teaching models to explicitly verbalize reward hacking behavior before RL significantly improves their detection, offering a practical path toward more transparent and safe AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
