通过纠正干预实现大推理模型的安全推理
Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention
AI总结：
本文提出IPO方法，通过干预优化提升大推理模型的安全性，显著降低有害性并保持性能。
摘要：
尽管大型推理模型（LRMs）在解决复杂问题方面取得了进展，但其思维链（CoT）推理过程中往往包含有害内容，即使最终响应看起来安全，这些内容仍可能持续存在。我们显示，现有方法中仍然存在这一问题，这些方法忽视了安全推理的独特重要性，削弱了它们的可信度，并在应用中可能因不安全的推理被恶意用户利用而带来潜在风险。因此，本文将重点对齐推理本身的安全性，并探索过程监督作为解决方案。然而，仅奖励安全推理证明不足，因为存在低滚动多样性和有限的训练信号。为解决这一挑战，我们首先深入研究安全推理的特点，发现几个关键见解：1）安全推理通常由几个关键的安全触发步骤集中；2）合规提示与不安全的延续有强相关性；3）纠正干预能可靠地将不安全轨迹引导到更安全的轨迹。受此启发，我们提出了干预偏好优化（IPO），一种对齐方法，通过将合规步骤替换为安全触发器，并构建具有强信号的配对进行偏好学习。在禁闭突破和对抗性安全基准测试中的实验表明，IPO显著提高了整体安全性，既在推理又在响应方面均优于基于SFT和RL的基线方法，有害性减少了超过30%，同时在各种推理任务中保持了出色的性能。结果突显了对推理进行显式对齐的重要性，并提供了一条更安全LRMs的实用路径。
英文摘要：
Although Large Reasoning Models (LRMs) have progressed in solving complex problems, their chain-of-thought (CoT) reasoning often contains harmful content that can persist even when the final responses appear safe. We show that this issue still remains in existing methods which overlook the unique significance of safe reasoning, undermining their trustworthiness and posing potential risks in applications if unsafe reasoning is accessible for and exploited by malicious users. We therefore shift our focus to aligning the safety of reasoning itself in this paper and explore process supervision as the solution. However, simply rewarding safe reasoning proves inadequate due to low rollout diversity and limited training signals. To tackle this challenge, we first delve into the characteristics of safe reasoning and uncover several critical insights that 1) safe reasoning is often consolidated by a few critical steps of safety triggers; 2) compliance cues strongly correlate with unsafe continuations; and 3) corrective interventions reliably steer unsafe trajectories towards safer traces. Motivated by these, we propose Intervened Preference Optimization (IPO), an alignment method that enforces safe reasoning by substituting compliance steps with safety triggers and constructing pairs for preference learning with strong signals. Experiments on jailbreak and adversarial safety benchmarks demonstrate that IPO remarkably improves overall safety regarding both reasoning and responses, outperforming SFT-based and RL-based baselines with a relative reduction of over 30% in harmfulness, while preserving excellent performance across diverse reasoning tasks. The results highlight the importance of explicit alignment for reasoning and provide a practical path to safer LRMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
