通过自适应安全上下文学习缓解LLM对齐中的安全性与效用权衡
Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning
AI总结：
本文提出ASCL框架，通过多轮工具使用过程提升推理能力，引入IFPO平衡优势估计，实现更高的整体性能。
摘要：
尽管推理模型在复杂推理任务中取得显著成功，但其不断增强的能力要求严格的安全部署。对于安全对齐，核心挑战在于安全与效用之间的固有权衡。然而，现有对齐策略通常通过上下文蒸馏显式构建CoT训练数据，这种做法无意中限制了推理能力，通过创建规则记忆与拒绝之间的刚性关联。为缓解安全-效用权衡，我们提出了自适应安全上下文学习（ASCL）框架，以在适当上下文中提高推理能力。ASCL将安全对齐建模为多轮工具使用过程，使模型能够自主决定何时咨询安全规则以及如何生成持续推理。此外，为对抗强化学习中对规则咨询的偏好，我们引入逆频率策略优化（IFPO）以重新平衡优势估计。通过解耦规则检索与后续推理，我们的方法在整体性能上优于基线方法。我们的代码可在https://github.com/ybwang119/ASCL公开获取。
英文摘要：
While reasoning models have achieved remarkable success in complex reasoning tasks, their increasing power necessitates stringent safety measures. For safety alignment, the core challenge lies in the inherent trade-off between safety and utility. However, prevailing alignment strategies typically construct CoT training data with explicit safety rules via context distillation. This approach inadvertently limits reasoning capabilities by creating a rigid association between rule memorization and refusal. To mitigate the safety-utility trade-off, we propose the Adaptive Safe Context Learning~(ASCL) framework to improve the reasoning given proper context. ASCL formulates safety alignment as a multi-turn tool-use process, empowering the model to autonomously decide when to consult safety rules and how to generate the ongoing reasoning. Furthermore, to counteract the preference for rule consultation during RL, we introduce Inverse Frequency Policy Optimization~(IFPO) to rebalance advantage estimates. By decoupling rule retrieval and subsequent reasoning, our method achieves higher overall performance compared to baselines. Our code is publicly available at https://github.com/ybwang119/ASCL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
