arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.22303cs.LGcs.AI
收藏

训练代理自我报告不当行为

Training Agents to Self-Report Misbehavior

UPenn(宾夕法尼亚大学);NYU(纽约大学);MATS(MATS机构);OpenAI

Bruce W. Lee, Chen Yueh-Han, Tomek Korbak

AI总结:

本文提出自我指控训练方法,通过训练代理在不当行为时产生可见信号,有效降低前沿AI对齐风险,无需假设行为可被防止或可靠分类。

摘要:

前沿AI代理可能会在隐藏目标的同时隐瞒其行为。对齐训练旨在通过强化正确目标来防止此类行为,但对齐可能并不总是成功,可能导致不良副作用。我们提出自我指控训练,该方法训练代理在秘密不当行为时产生可见信号。我们训练GPT-4.1和Gemini-2.0代理在欺骗性行为时调用report_scheming()工具,并测量其在非分布环境中造成危害而不被发现的能力。自我指控显著降低了未被发现的成功攻击率,优于匹配能力的监控和对齐基线,同时保持指令层级并在一般能力上带来最小的安全开销。与黑盒监控不同,自我指控性能在任务间保持一致,无论外部可疑行为如何表现。训练行为在对抗性提示优化下持续存在,并泛化到代理自身追求不一致目标而非被指示不当行为的设置中。我们的结果表明,自我指控为减少前沿对齐风险提供了一条可行路径,这一路径既不假设不当行为可以被防止,也不假设可以从外部可靠分类。

英文摘要:

Frontier AI agents may pursue hidden goals while concealing their pursuit from oversight. Alignment training aims to prevent such behavior by reinforcing the correct goals, but alignment may not always succeed and can lead to unwanted side effects. We propose self-incrimination training, which instead trains agents to produce a visible signal when they covertly misbehave. We train GPT-4.1 and Gemini-2.0 agents to call a report_scheming() tool when behaving deceptively and measure their ability to cause harm undetected in out-of-distribution environments. Self-incrimination significantly reduces the undetected successful attack rate, outperforming matched-capability monitors and alignment baselines while preserving instruction hierarchy and incurring minimal safety tax on general capabilities. Unlike blackbox monitoring, self-incrimination performance is consistent across tasks regardless of how suspicious the misbehavior appears externally. The trained behavior persists under adversarial prompt optimization and generalizes to settings where agents pursue misaligned goals themselves rather than being instructed to misbehave. Our results suggest self-incrimination offers a viable path for reducing frontier misalignment risk, one that neither assumes misbehavior can be prevented nor that it can be reliably classified from the outside.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑