arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.29601cs.CLcs.AIcs.LG
收藏

训练审慎监控器用于黑箱策划检测

Training Deliberative Monitors for Black-Box Scheming Detection

Independent(独立);MATS Research(MATS研究);Astra Fellowship;Apollo Research(Apollo研究)

Aditya Sinha, Akshat Naik, Victor Gillioz, Simon Storf, Kilian Merkelbach, Rich Barton-Cooper, Axel Højmark, Marius Hobbhahn

AI总结:

提出一种基于行动轨迹的审慎监控方法,通过蒸馏前沿模型的推理过程训练开源模型,以低成本高精度检测智能体的策划与破坏行为。

摘要:

随着自主智能体在执行现实任务方面变得愈发强大,区分策划行为与良性任务追求可能成为AI控制的核心问题。现有监控器通常依赖思维链访问或内部激活,或使用提示的前沿模型,这些在部署中可能不可用、不可靠或成本高昂。在本工作中,我们研究仅基于行动的审慎监控器:较小的开源模型,经过训练可从智能体轨迹中检测策划与破坏行为,而无需访问被监控智能体的推理或模型内部。我们的方法受审慎对齐启发,使用策划规范从前沿教师模型中引出结构化推理,通过独立的评判器进行过滤,并通过监督微调和强化学习将最高质量的推理蒸馏到开源监控器中。我们在五个数据集上训练,并在六个分布外智能体失调基准上评估。我们表明,将我们的方法应用于Qwen3.5-27B,其性能优于所有低成本前沿模型作为提示监控器(Gemini 3.1 Flash-Lite、GPT-5.4 Nano和Claude Haiku 4.5)以及Gemini 2.5 Pro,同时实现了更低的边际推理成本(每1000次评估的token计费美元)。更强的提示前沿监控器(Gemini 3.1 Pro、GPT-5.4、Claude Sonnet 4.6和Claude Opus 4.6)实现了更高的性能,但边际推理成本大约高出16-34倍。我们训练的多个监控器在我们评估的监控器中位于经验成本-性能帕累托前沿,为提示前沿模型提供了实用的低成本、低误报率替代方案。

英文摘要:

As autonomous agents become more capable of performing real-world tasks, distinguishing scheming behavior from benign task pursuit may become a central AI control problem. Existing monitors often rely on chain-of-thought access or internal activations, or use prompted frontier models, all of which can be unavailable, unreliable or expensive in deployment. In this work, we study action-only deliberative monitors: smaller open-weight models trained to detect scheming and sabotage from agentic trajectories without accessing the monitored agent's reasoning or model internals. Our method, inspired by deliberative alignment, uses a scheming specification to elicit structured rationales from a frontier teacher, filters them with a separate judge, and distills the highest-quality rationales into open-weight monitors with supervised fine-tuning and reinforcement learning. We train on five datasets, and evaluate across six out-of-distribution agentic misalignment benchmarks. We show that applying our method to Qwen3.5-27B yields higher performance than all low-cost frontier models as prompted monitors (Gemini 3.1 Flash-Lite, GPT-5.4 Nano, and Claude Haiku 4.5) and than Gemini 2.5 Pro, while also achieving lower marginal inference cost (token-metered USD per 1,000 evaluations). Stronger prompted frontier monitors (Gemini 3.1 Pro, GPT-5.4, Claude Sonnet 4.6, and Claude Opus 4.6) achieve higher performance but at roughly $16$--$34\times$ higher marginal inference cost. Several of our trained monitors are positioned on the empirical cost--performance Pareto frontier among the monitors we evaluate, providing practical low-cost, low-FPR alternatives to prompted frontier models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑