arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.23488cs.LG
收藏

合成轨迹反映真实的奖励黑客行为吗?对监控真实世界代码生成中黑客行为的系统研究

Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generation

Peking University(北京大学);University of California, Los Angeles(加州大学洛杉矶分校);Arena;University of Maryland(马里兰大学);Mohamed bin Zayed University of Artificial Intelligence(莫莫迪 bin Zayed 人工智能大学)

Lichen Li, Hengguang Zhou, Yijun Liang, Tianyi Zhou, Cho-Jui Hsieh

AI总结:

本文研究合成轨迹与真实世界中代码生成奖励黑客行为的差异,通过对比监控器在合成与真实数据上的表现,发现合成数据训练的监控器无法泛化到真实黑客行为,而真实数据训练的监控器对未见过的黑客类型更具泛化能力。

摘要:

在代码生成中,奖励黑客行为是指模型利用评估漏洞获得完整奖励而未正确解决问题,这对强化学习(RL)和推理模型的部署构成重大挑战。现有研究主要集中在合成黑客轨迹上,但这些合成行为是否能真实反映野外自然出现的黑客行为仍不明确。本文系统分析了合成与野外奖励黑客行为的差异。我们考察了通过提示诱导的黑客行为与RL训练期间出现的行为有多大相似性,并检查在合成轨迹上训练的监控器是否能泛化到自然出现但此前未见过的黑客行为。为了扩大野外奖励黑客轨迹的整理规模,我们修改了群相对策略优化(GRPO),通过注入冲突的单元测试作为追踪器,并应用“重采样直到黑客”的机制。通过在合成与野外数据上训练的监控器之间的受控比较,我们发现(1)合成数据训练的监控器无法泛化到“野外”黑客行为,(2)在我们“野外”轨迹上训练的监控器对未见过的黑客类型具有更强的泛化能力。我们的结果表明,合成奖励黑客数据可能无法完全反映自然奖励黑客行为,仅依赖合成数据可能导致误导性结论。代码库可在 https://github.com/LichenLillc/CoTMonitoring.git 获取。

英文摘要:

Reward hacking in code generation, where models exploit evaluation loopholes to obtain high reward without correctly solving the intended task, poses a critical challenge for Reinforcement Learning (RL) and the deployment of reasoning models. Existing studies often rely on explicitly prompted hacking trajectories, but it remains unclear whether monitors trained on such data can detect reward hacks that arise without direct hacking instructions during RL training. In this work, we introduce Trace-and-Amplify, a framework for scalable curation of reward-hacking trajectories that arise during RL training without explicit hacking instructions. The framework uses unit-test tracers to identify hacking solutions when they occur and retains such trajectories for monitor training and evaluation. Through controlled comparisons between monitors trained on prompt-elicited hacking trajectories and training-time reward-hacking trajectories collected by Trace-and-Amplify, we find that \textbf{(1) prompt-elicited-data-trained monitors often fail to generalize to trajectories curated by our framework}, and \textbf{(2) monitors trained on our Trace-and-Amplify trajectories demonstrate stronger generalizability to unseen hacking types}. Our results indicate that prompted reward hacking data may not fully reflect training-time reward-hacking behaviors, and that relying solely on these data can lead to misleading conclusions. Codebase is available at https://github.com/LichenLillc/CoTMonitoring.git

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑