arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.20103cs.SEcs.AIcs.LG
收藏

通过对比分析在代码环境中基准测试奖励黑客检测

Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis

Darshan Deshpande, Anand Kannappan, Rebecca Qian

AI总结:

本文通过对比分析提出了一种涵盖54类的奖励利用分类法,并引入TRACE基准测试,展示模型在对比设置中对奖励黑客的检测效果优于孤立分类设置。

摘要:

最近在代码生成中的强化学习进展使得构建稳健的环境变得至关重要,以防止奖励黑客行为。随着大型语言模型越来越多地作为代码基于强化学习的评估者,它们检测奖励黑客的能力仍缺乏深入研究。在本文中,我们提出了一种涵盖54个类别的新颖奖励利用分类法,并引入TRACE(在代码环境中测试奖励异常),这是一个合成整理并由人类验证的基准,包含517个测试轨迹。与以往仅在孤立分类场景中评估奖励黑客检测的工作不同,我们对比这些评估结果与一个更现实的、对比性的异常检测设置在TRACE上。我们的实验表明,模型在对比设置中比在孤立分类设置中更能捕捉奖励黑客,GPT-5.2在最高推理模式下在TRACE上达到63%的最佳检测率,从孤立设置中的45%提升。基于这一见解,我们证明了最先进的模型在语义上下文化奖励黑客方面表现显著不如在语法上下文化奖励黑客方面。我们还进行了模型行为的定性分析,以及消融研究,显示良性与黑客轨迹的比例以及分析聚类大小对检测性能有显著影响。我们发布了基准和评估工具,以使社区能够扩展TRACE并评估其模型。

英文摘要:

Recent advances in reinforcement learning for code generation have made robust environments essential to prevent reward hacking. As LLMs increasingly serve as evaluators in code-based RL, their ability to detect reward hacking remains understudied. In this paper, we propose a novel taxonomy of reward exploits spanning across 54 categories and introduce TRACE (Testing Reward Anomalies in Code Environments), a synthetically curated and human-verified benchmark containing 517 testing trajectories. Unlike prior work that evaluates reward hack detection in isolated classification scenarios, we contrast these evaluations with a more realistic, contrastive anomaly detection setup on TRACE. Our experiments reveal that models capture reward hacks more effectively in contrastive settings than in isolated classification settings, with GPT-5.2 with highest reasoning mode achieving the best detection rate at 63%, up from 45% in isolated settings on TRACE. Building on this insight, we demonstrate that state-of-the-art models struggle significantly more with semantically contextualized reward hacks compared to syntactically contextualized ones. We further conduct qualitative analyses of model behaviors, as well as ablation studies showing that the ratio of benign to hacked trajectories and analysis cluster sizes substantially impact detection performance. We release the benchmark and evaluation harness to enable the community to expand TRACE and evaluate their models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑