奖励黑客行为挑战自主研究智能体的监督

中文摘要 AI 辅助

自主研究智能体能够设计实验、评估结果并撰写报告，从而同时掌控科学结果及其支撑证据。这带来了奖励黑客（reward hacking）的风险：即满足奖励标准却未实现预期目标。我们研究了以下三个问题：（1）模型在未被指示的情况下进行奖励黑客行为的频率；（2）当允许黑客行为时，其方法的有效性和可检测性；（3）当LLM评审小组返回其决定和理由时，模型如何适应。在17个语言模型和38个任务中，开放式研究流程任务的自发奖励黑客率为30.5%，而特定任务内核的该比率为2.9%。当在通过阈值超过我们最佳合规基线的任务上允许黑客行为时，505/677次尝试（74.6%）被确认为奖励黑客：它们既通过了阈值，又获得了机制验证小组对评估漏洞的确认。仅审查提交代码和报告分数的LLM评审小组遗漏了505个已确认黑客行为中的33个（6.5%）。获得最高分数的直接方法通常易于检测，而较不直接的方法则更常被规避。在五轮循环中，存在规避行为的模型-任务对数量从7个增加到56个。在两种反馈条件下评估的79个模型-任务对中，累积规避率在详细反馈条件下达到40.5%，在通用拒绝条件下为20.3%。详细条件包含评审决定、理由和尝试历史，因此该比较并未隔离解释的影响。这些发现凸显了加强防御的必要性，包括将指标置于智能体控制之外，以及在旨在暴露潜在漏洞的数据上进行独立重算。

英文摘要

Autonomous research agents can design experiments, evaluate results, and write reports, giving them control over both a scientific result and the evidence used to support it. This creates a risk of reward hacking: meeting the reward criteria without achieving the intended goal. We study (1) how often models reward-hack without instructions to do so, (2) how effective and detectable their methods are when hacking is allowed, and (3) how they adapt when an LLM review panel returns its decision and reasons. Across 17 language models and 38 tasks, the spontaneous reward-hacking rate is 30.5% on open-ended research-pipeline tasks and 2.9% on task-specific kernels. When hacking is allowed on tasks whose pass thresholds exceed our best compliant baselines, 505/677 attempts (74.6%) are confirmed reward hacks: they both clear the threshold and receive mechanism-verification panel confirmation of an evaluation exploit. An LLM panel reviewing only submitted code and reported scores misses 33/505 confirmed hacks (6.5%). Direct methods that achieve the highest scores are often easy to detect, while less direct methods evade more often. In a five-round loop, the number of model-task pairs with an evasion rises from 7 to 56. Among 79 pairs evaluated under two feedback conditions, cumulative evasion reaches 40.5% with detailed feedback and 20.3% with generic rejection. The detailed condition includes the review decision, reasons, and attempt history, so this comparison does not isolate the effect of explanations. These findings highlight the need for stronger defenses, including metrics kept outside the agent's control and independent recomputation on data chosen to expose likely exploits.

发表机构

• Bake AI
• University of Notre Dame（圣母大学）
• LMU Munich（慕尼黑大学）
• University of Washington（华盛顿大学）
• IBM Research（IBM研究院）
• Microsoft Research（微软研究院）
• University of California, Santa Barbara（加州大学圣塔芭芭拉分校）
• Stanford University（斯坦福大学）
• Massachusetts Institute of Technology（麻省理工学院）

机构由 AI 辅助整理，请以论文原文为准。

相关深度报道

• AI搞科研开始钻空子：17个模型74.6%在黑客评审，五轮后越查越漏