AdvChain：面向大型推理模型鲁棒安全对齐的对抗性思维链微调
AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models
AI总结：
针对安全CoT微调中的“雪球效应”，本文提出AdvChain对抗性思维链微调范式，通过诱惑—纠正与犹豫—纠正样本训练模型自我纠正，提升抗攻击能力并降低过度拒绝。
摘要：
大型推理模型（LRM）已通过思维链（CoT）推理在复杂问题求解中展现出卓越能力。然而，CoT的多步骤特性带来了超越传统语言模型对齐的新安全挑战。我们发现当前安全CoT微调方法存在一种失效模式：“雪球效应”，即轻微的推理偏差会在思维过程中逐步放大，最终导致有害遵从或过度拒绝。该效应源于模型被训练为模仿完美推理脚本，却没有学会自我纠正。为解决这一局限，我们提出AdvChain，这是一种通过对抗性CoT微调教授模型动态自我纠正的对齐范式。我们的方法包括构建一个包含“诱惑—纠正”和“犹豫—纠正”样本的数据集，使模型学习从有害推理漂移和不必要谨慎中恢复。大量实验表明，AdvChain显著增强了模型抵御越狱攻击和CoT劫持的鲁棒性，同时大幅降低了对良性提示的过度拒绝，在不损害推理能力的情况下实现了更优的安全—效用平衡。我们的工作为构建更鲁棒、更可靠的推理模型确立了新方向。
英文摘要：
Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in complex problem-solving through Chain-of-Thought (CoT) reasoning. However, the multi-step nature of CoT introduces new safety challenges that extend beyond conventional language model alignment. We identify a failure mode in current safety CoT tuning methods: the \textit{snowball effect}, where minor reasoning deviations progressively amplify throughout the thought process, leading to either harmful compliance or excessive refusal. This effect stems from models being trained to imitate perfect reasoning scripts without learning to self-correct. To address this limitation, we propose AdvChain, an alignment paradigm that teaches models dynamic self-correction through adversarial CoT tuning. Our method involves constructing a dataset containing Temptation-Correction and Hesitation-Correction samples, where models learn to recover from harmful reasoning drifts and unnecessary cautions. Extensive experiments show that AdvChain significantly enhances robustness against jailbreak attacks and CoT hijacking while substantially reducing over-refusal on benign prompts, achieving a superior safety-utility balance without compromising reasoning capabilities. Our work establishes a new direction for building more robust and reliable reasoning models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
