双因子偏好优化：平衡语言模型中的安全性与有用性
Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models
AI总结：
本文提出双因子偏好优化（BFPO）监督学习框架，将安全性与有用性的联合RLHF目标转化为单一监督目标，以更低计算和人工成本实现两者平衡并显著超越现有方法。
摘要：
通过人类反馈强化学习（RLHF）在人类偏好上微调大语言模型（LLMs）已被证明能够有效提升模型能力。然而，在微调过程中确保LLMs的安全性仍然是一个关键问题，并且在RLHF中缓解安全性与有用性之间的潜在冲突成本高昂。为解决这一问题，我们提出了一种名为双因子偏好优化（BFPO）的监督学习框架，该框架将安全性与有用性的联合RLHF目标重新参数化为单一的监督学习目标。在监督优化中，使用一个标注函数来捕捉全局偏好排序，以同时平衡安全性与有用性。为评估BFPO，我们开发了一个基准，其中包含针对有用性和无害性的全面判别性与生成性任务。结果表明，我们的方法在安全性和有用性方面均显著优于现有方法。此外，BFPO在不到10%的计算资源以及人工提示和标注过程的情况下，达到了与严重依赖人工劳动的方法相同的安全水平。训练配方可在此获取：https://github.com/wx-zhang/bfpo。
英文摘要：
Fine-tuning large language models (LLMs) on human preferences, typically through reinforcement learning from human feedback (RLHF), has proven successful in enhancing their capabilities. However, ensuring the safety of LLMs during fine-tuning remains a critical concern, and mitigating the potential conflicts in safety and helpfulness is costly in RLHF. To address this issue, we propose a supervised learning framework called Bi-Factorial Preference Optimization (BFPO), which re-parameterizes a joint RLHF objective of both safety and helpfulness into a single supervised learning objective. In supervised optimization, a labeling function is used to capture the global preferences ranking to balance both safety and helpfulness. To evaluate BFPO, we develop a benchmark that includes comprehensive discriminative and generative tasks for helpfulness and harmlessness. The results indicate that our method significantly outperforms existing approaches in both safety and helpfulness. Moreover, BFPO achieves the same level of safety as methods that heavily rely on human labor with less than 10\% of the computational resources and human prompting and annotation process. The training recipes can be found here: https://github.com/wx-zhang/bfpo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
