发表机构
University of Washington（华盛顿大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究证明，在代码生成环境中，低监控读数无法证明行为控制，因为探针和前缀训练分数可能因位置不匹配或延迟承诺而偏低，需进行带外行为检查。
AI 中文摘要
使用可验证奖励进行后训练可能引发奖励黑客行为，这促使在训练目标中引入监控器，而非仅用于离线审计。我们证明，较低的监控读数无法识别此类干预是否控制了行为。在一个主要漏洞在推理轨迹开始时即可利用的代码生成环境中，我们针对三个通过相同离线门槛的监控器训练策略：一个域内激活探针，以及两个基于策略过早承诺自身最终答案的惩罚项。探针分数从第一个记录的训练步骤起便处于其数值下限，而每个前缀训练运行在终点时的训练分数中位数为零。这些读数估计不同的量，我们不比较其尺度；然而，在每个监控器家族内部，低值并不能确立行为控制。在一个固定配置内，具有相同零中位数训练分数的前缀训练运行，仅因随机种子不同，便从低黑客份额的混合机制变化到近乎纯奖励黑客。所有探针运行均达到黑客机制，但其下限读数反映了探针验证位置与训练期间读取位置之间的不匹配，而非这种模糊性的第二次实例。文本级分析识别出一种前缀失败模式：通用规划和填充外壳将漏洞推迟到截断之后，但并未将其从最终输出中消除。因此，低测量承诺无法区分低黑客份额与对漏洞的延迟承诺。离线区分和低监控对齐读数不足以作为行为控制的证据；需要进行带外行为检查。我们表征终点读数，而非其演变过程。代码可在该 https URL 获取。
英文摘要
Post-training with verifiable rewards can induce reward hacking, motivating the use of monitors within the training objective rather than solely for offline auditing. We show that a low monitor readout does not identify whether such an intervention controls behavior. In a code-generation environment whose dominant exploit is available at the start of the reasoning trace, we train policies against three monitors that pass the same offline gate: an in-domain activation probe and two penalties conditioned on how early the policy commits to its own final answer. The probe score is at its numerical floor from the first recorded training step, and the trained-score median is zero for every prefix-trained run at the endpoint. These readouts estimate different quantities, and we do not compare their scales; within each monitor family, however, low values do not establish behavioral control. Within one fixed configuration, prefix-trained runs with the same zero-median trained score range, by seed alone, from a mixed regime with a low hacking share to near-pure reward hacking. All probe runs reach the hacking regime, but their floor-level readout reflects a mismatch between the position where the probe was validated and the position where it was read during training, not a second instance of this ambiguity. Text-level analysis identifies a prefix failure mode: generic planning and filler shells postpone the exploit past the cut without eliminating it from the final output. Low measured commitment therefore does not distinguish a low hacking share from delayed commitment to the exploit. Offline discrimination and low monitor-aligned readouts are insufficient evidence of behavioral control; an out-of-band behavioral check is required. We characterize the endpoint readout, not its evolution. Code is available at https://github.com/zhezhou1106/spoof-cost.
Comments17 pages, 2 figures, 10 tables. Accepted as a poster at the NeurIPS 2026 Workshop on Foundations of LLM Post-Training in Changing Environments (FLLMPT)
