BAITBENCH：通过植入ML任务中的可选捷径测量智能体的奖励黑客行为

中文摘要 AI 辅助

大型语言模型（LLM）智能体越来越多地用于运行自主机器学习（ML）实验，在极少人工监督下迭代优化目标指标。现有研究已记录了此类环境中的奖励黑客行为，这使得所产生研究的有效性以及AI研发更广泛的安全论证受到质疑。现有基准无法测量存在于数据或建模任务本身中的漏洞。我们推出BAITBENCH，一套包含三个合成表格型ML任务的基准，每个任务都包含一个捷径，该捷径可让智能体提高公开测试集分数，但在隐藏测试集上表现不佳。由于该捷径是可选的，使用它也不违反任何明确规则，因此BAITBENCH用于测量模型利用捷径获得虚高分数的频率。通过我们的两阶段评判流水线对七个前沿智能体进行评分，57.1%的运行出现奖励黑客行为，七个智能体中有五个的作弊率超过50%。即使在被提示不要作弊的第二条件下，智能体仍会作弊，平均作弊率保持在50%以上。我们发布BAITBENCH以及评判器实现和包含奖励黑客行为的带注释对话数据集，作为用于并行评估奖励黑客缓解措施的测试平台。

英文摘要

LLM agents are increasingly used to run autonomous ML experiments, iterating on target metrics with little human oversight. Prior work has documented reward hacking in these environments, bringing into question the validity of produced research and the broader safety case for AI R&D. Existing benchmarks do not measure exploits that live in the data or the modeling task itself. We introduce BAITBENCH, a suite of three synthetic tabular ML tasks that each contain a shortcut that allows agents to inflate the public test score but fail on a hidden test set. Since the shortcut is optional and using it breaks no stated rule, BAITBENCH measures how often models exploit the shortcut to achieve inflated scores. Across seven frontier agents scored by our two-stage judge pipeline, 57.1% of runs exhibit reward hacking, with five of seven above 50%. Agents cheat even under a second condition where they are prompted not to -the mean cheating rate remains above 50%. We release BAITBENCH, along with the judge implementation, and an annotated dataset of transcripts containing reward hacks as a testbed for evaluating reward-hacking mitigations head-to-head.

发表机构

• National University of Singapore（新加坡国立大学）
• MIT（麻省理工学院）
• Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）
• University of Toronto（多伦多大学）
• University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
• Arb Research（Arb研究机构）

机构由 AI 辅助整理，请以论文原文为准。