通过约束式直接偏好优化增强 LLM 安全性
Enhancing LLM Safety via Constrained Direct Preference Optimization
AI总结：
本文提出 C-DPO，将对偶梯度下降与 DPO 结合，在不使用强化学习的情况下平衡 LLM 的有用性与无害性，并在相同安全约束下优于安全 RLHF。
摘要：
大型语言模型（LLM）能力的快速提升，使得让 AI 系统与多样化人类偏好保持一致成为迫切需求，从而同时增强其有用性和安全性，尽管这些目标往往相互冲突。为解决这一重要问题，一种有前景的方法是在微调阶段通过约束式“基于人类反馈的强化学习”（RLHF）框架施加安全约束。然而，该方法计算成本高昂且通常不稳定。在本文中，我们提出 Constrained DPO（C-DPO），它是近期提出的用于微调 LLM 的 Direct Preference Optimization（DPO，直接偏好优化）方法的一种新颖扩展，兼具高效性与轻量性。通过将对偶梯度下降与 DPO 相结合，我们的方法无需使用强化学习，即可在有用性与无害性之间找到近乎最优的权衡。实验表明，我们的方法为 LLM 提供了 DPO 所缺失的安全保证；同时，在相同安全约束下，与近期提出的安全 RLHF 方法相比，其获得了显著更高的奖励。警告：本文包含可能具有冒犯性或有害性的示例数据。
英文摘要：
The rapidly increasing capabilities of large language models (LLMs) raise an urgent need to align AI systems with diverse human preferences to simultaneously enhance their usefulness and safety, despite the often conflicting nature of these goals. To address this important problem, a promising approach is to enforce a safety constraint at the fine-tuning stage through a constrained Reinforcement Learning from Human Feedback (RLHF) framework. This approach, however, is computationally expensive and often unstable. In this work, we introduce Constrained DPO (C-DPO), a novel extension of the recently proposed Direct Preference Optimization (DPO) approach for fine-tuning LLMs that is both efficient and lightweight. By integrating dual gradient descent and DPO, our method identifies a nearly optimal trade-off between helpfulness and harmlessness without using reinforcement learning. Empirically, our approach provides a safety guarantee to LLMs that is missing in DPO while achieving significantly higher rewards under the same safety constraint compared to a recently proposed safe RLHF approach. Warning: This paper contains example data that may be offensive or harmful.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
