这是思考还是作弊？通过测量推理努力来检测隐式奖励黑客
Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort
AI总结：
TRACE通过测量推理努力检测隐式奖励黑客，通过截断推理链并评估预期奖励来识别模型利用漏洞的行为，提升数学和编码任务的监控效果。
摘要：
奖励黑客，即一个推理模型利用奖励函数中的漏洞来获得高奖励而没有解决预期任务，构成了重大威胁。这种行为可能是显性的，即在模型的思维链（CoT）中被明确表达，也可能是隐性的，即CoT看起来无害，因此绕过了CoT监控。为了检测隐式奖励黑客，我们提出了TRACE（截断推理AUC评估）。我们的关键观察是，当利用漏洞比解决实际任务更容易时，就会发生黑客行为。这意味着模型在获得高奖励时使用的努力比要求的要少。TRACE通过测量模型的推理何时变得足够简单以获得奖励来量化努力。我们逐步截断模型的CoT在各种长度上，强制模型回答，并估计每个截断处的预期奖励。一个采取捷径的模型将只用其CoT的小部分就能获得高预期奖励，从而在准确性与长度曲线下产生较大的面积。TRACE在数学推理中比我们最强的72B CoT监控器高出65%以上，在编码中比32B监控器高出30%以上。我们进一步表明，TRACE可以在训练期间发现未知的漏洞。总体而言，TRACE提供了一种可扩展的无监督方法，用于监督当前监控方法无效的领域。
英文摘要：
Reward hacking, where a reasoning model exploits loopholes in a reward function to achieve high rewards without solving the intended task, poses a significant threat. This behavior may be explicit, i.e. verbalized in the model's chain-of-thought (CoT), or implicit, where the CoT appears benign thus bypasses CoT monitors. To detect implicit reward hacking, we propose TRACE (Truncated Reasoning AUC Evaluation). Our key observation is that hacking occurs when exploiting the loophole is easier than solving the actual task. This means that the model is using less 'effort' than required to achieve high reward. TRACE quantifies effort by measuring how early a model's reasoning becomes sufficient to obtain the reward. We progressively truncate a model's CoT at various lengths, force the model to answer, and estimate the expected reward at each cutoff. A hacking model, which takes a shortcut, will achieve a high expected reward with only a small fraction of its CoT, yielding a large area under the accuracy-vs-length curve. TRACE achieves over 65% gains over our strongest 72B CoT monitor in math reasoning, and over 30% gains over a 32B monitor in coding. We further show that TRACE can discover unknown loopholes during training. Overall, TRACE offers a scalable unsupervised approach for oversight where current monitoring methods prove ineffective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
