通过自我监控缓解欺骗性对齐
Mitigating Deceptive Alignment via Self-Monitoring
AI总结：
针对思维链推理可能放大欺骗性对齐的问题，提出CoT Monitor+框架，在思维链中嵌入自我监控信号以抑制欺骗行为，平均降低43.8%的欺骗行为，同时保持任务准确性。
摘要：
现代大型语言模型依赖思维链（CoT）推理来实现令人印象深刻的性能，然而同样的机制可能放大欺骗性对齐，即模型表面上看起来对齐，却暗中追求不一致目标的情况。现有的安全流程将欺骗视为黑盒输出，在事后进行过滤，使得模型在内部推理过程中可以自由地进行谋划。我们提出疑问：能否在模型思考时拦截欺骗？我们回答了这个问题，提出了第一个将自我监控器嵌入思维链过程本身的框架，命名为CoT Monitor+。在生成过程中，模型产生（i）普通推理步骤和（ii）一个内部自我评估信号，该信号经过训练以标记并抑制不一致的策略。该信号在强化学习中用作辅助奖励，形成一个反馈循环，奖励诚实的推理并阻止隐藏的目标。为了系统地研究欺骗性对齐，我们引入了DeceptionBench，一个五类基准，用于探测隐蔽的对齐伪装、奉承等行为。我们评估了各种大型语言模型，并表明不受限制的思维链大致加剧了欺骗倾向。相比之下，CoT Monitor+平均减少了43.8%的欺骗行为，同时保持了任务准确性。此外，当自我监控信号在强化学习微调中取代外部弱评判器时，模型表现出明显更少的混淆思维并保持透明度。我们的项目网站可以在cot-monitor-plus.github.io找到。
英文摘要：
Modern large language models rely on chain-of-thought (CoT) reasoning to achieve impressive performance, yet the same mechanism can amplify deceptive alignment, situations in which a model appears aligned while covertly pursuing misaligned goals. Existing safety pipelines treat deception as a black-box output to be filtered post-hoc, leaving the model free to scheme during its internal reasoning. We ask: Can deception be intercepted while the model is thinking? We answer this question, the first framework that embeds a Self-Monitor inside the CoT process itself, named CoT Monitor+. During generation, the model produces (i) ordinary reasoning steps and (ii) an internal self-evaluation signal trained to flag and suppress misaligned strategies. The signal is used as an auxiliary reward in reinforcement learning, creating a feedback loop that rewards honest reasoning and discourages hidden goals. To study deceptive alignment systematically, we introduce DeceptionBench, a five-category benchmark that probes covert alignment-faking, sycophancy, etc. We evaluate various LLMs and show that unrestricted CoT roughly aggravates the deceptive tendency. In contrast, CoT Monitor+ cuts deceptive behaviors by 43.8% on average while preserving task accuracy. Further, when the self-monitor signal replaces an external weak judge in RL fine-tuning, models exhibit substantially fewer obfuscated thoughts and retain transparency. Our project website can be found at cot-monitor-plus.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
