arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.17596cs.CRcs.AI
收藏

终端 wrench:331个可奖励黑客环境和3,632条exploit轨迹的数据集

Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories

Fewshot Corp(Fewshot公司);Carnegie Mellon University(卡内基梅隆大学)

Ivan Bercovich, Ivgeni Segal, Kexun Zhang, Shashwat Saxena, Aditi Raghunathan, Ziqian Zhong

AI总结:

本文发布了一个包含331个终端代理基准环境和3,632条exploit轨迹的数据集,展示了不同任务中的特定exploit方法,并通过LLM评估检测性能下降。

摘要:

我们发布了Terminal Wrench,一个由331个终端代理基准环境子集构成的数据集,源自流行的开源基准测试,这些测试已被证明是可奖励黑客的。数据集包括3,632条黑客轨迹和2,352条合法基线轨迹,涵盖三个前沿模型(Claude Opus 4.6、Gemini 3.1 Pro、GPT-5.4)。每个条目保留了原始任务定义及展示如何绕过验证者的完整攻击轨迹。还包含未按预期解决的任务案例。任务涵盖系统管理、机器学习、软件工程和安全挑战;exploit范围从简单的输出欺骗到栈帧 introspection、标准库修补和rootkit风格的二进制劫持。关键的是,这些exploit特定于每个任务,而非评估工具,使其更难修复。我们还展示了黑客轨迹经过清洗或移除推理轨迹后,由LLM评估者评分,显示当移除链式思维时检测性能显著下降(AUC从0.97降至0.92)。数据集可在https://github.com/few-sh/terminal-wrench公开获取。

英文摘要:

We release Terminal Wrench, a subset of 331 terminal-agent benchmark environments, copied from the popular open benchmarks that are demonstrably reward-hackable. The data set includes 3,632 hack trajectories and 2,352 legitimate baseline trajectories across three frontier models (Claude Opus 4.6, Gemini 3.1 Pro, GPT-5.4). Each entry preserves the original task definition alongside full attack trajectories that show how the verifier was bypassed. It also includes cases where the task was not solved as intended. The tasks span system administration, machine learning, software engineering, and security challenges; the exploits range from simple output spoofing to stack-frame introspection, standard-library patching, and rootkit-style binary hijacking. Crucially, these exploits are specific to each task, rather than the evaluation harness, making them harder to patch. We also present a monitorability study in which hack trajectories are sanitized or stripped of reasoning traces and then scored by an LLM judge, showing that detection degrades meaningfully when chain-of-thought is removed (AUC drops from 0.97 to 0.92). The data set is publicly available at https://github.com/few-sh/terminal-wrench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑