arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.08066cs.AIcs.LG
收藏

说服攻击会降低思维链监测的有效性

Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring

LASR Labs(LASR实验室);University College London(伦敦大学学院);Google DeepMind(谷歌DeepMind)

Jennifer Za, Julija Bainiaksina, Nikita Ostrovsky, Tanush Chopra, Victoria Krakovna

AI总结:

研究 CoT 监测在对抗性说服攻击下的有效性,设计评估框架发现其单独使用不足以抵御攻击,引入事实核查监测框架,不同模型家族配对可有效减少违反政策行动批准,为 CoT 监测抗攻击提供有力缓解措施。

摘要:

思维链(CoT)监测是人工智能代理一种很有前景的安全机制,基于可见推理痕迹可揭示不一致或欺骗行为这一前提。虽然在标准场景中有效,但近期工作表明大语言模型仍易受基于说服的越狱攻击,即自然语言论证会 override 模型约束。我们对这种脆弱性是否扩展到监测大语言模型进行压力测试:对抗性代理能否说服其 CoT 监测器批准违反监测器策略的提议行动?我们设计了一个有 40 项任务的评估框架并分析数千次代理 - 监测器交互,其中代理被指示为违反政策的提议进行论证。我们发现,在这种对抗性设置中,监测器对代理 CoT 推理的访问平均增加而非减少对有害行动的批准达 9.5%,因为便签簿提供了额外的说服渠道。为解决此问题,我们引入了一个事实核查监测框架。我们发现,来自不同模型家族的事实核查器和监测器配对,例如一个 Claude 3.7 Sonnet 监测器与一个 GPT - 4.1 事实核查器配对,与在事实核查和监测角色都使用相同模型时仅 6%相比,可将违反政策行动的批准减少多达 45%。我们的结果表明,仅 CoT 监测可能不足以抵御对抗性说服,且模型多样化的事实核查提供了强有力的缓解措施。

英文摘要:

Chain-of-thought (CoT) monitoring is a promising safety mechanism for AI agents, based on the premise that visible reasoning traces can surface misaligned or deceptive behavior. While effective in standard scenarios, recent work highlights that LLMs remain vulnerable to persuasion-based jailbreaks, where natural-language arguments override model constraints. We stress-test whether this vulnerability extends to monitoring LLMs: can an adversarial agent persuade its CoT monitor to approve proposed actions that violate the monitor's policy? We design an evaluation framework with 40 tasks and analyze thousands of agent-monitor interactions, where agents are instructed to argue for policy-violating proposals. We find that in such adversarial settings, monitor access to the agent's CoT reasoning increases rather than decreases approval of harmful actions on average by 9.5%, as the scratchpad provides an additional persuasion channel. To address this, we introduce a fact-checking monitoring framework. We find that a fact-checker and monitor pairing from different model families, for example a Claude 3.7 Sonnet monitor paired with a GPT-4.1 fact-checker, reduces approval of policy-violating actions by up to 45%, compared to only 6%, when using the same model for both fact-checking and monitoring roles. Our results demonstrate that CoT monitoring alone may be insufficient against adversarial persuasion, and that model-diverse fact-checking provides a robust mitigation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑