优化安全且对齐的语言生成：一种多目标GRPO方法
Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach
AI总结：
本文提出一种多目标GRPO框架，结合多标签奖励回归模型，在0.5B至14B参数规模上实现安全且对齐的语言生成，以较低计算成本平衡安全性、有用性等目标，优于PPO和DPO。
摘要：
将大型语言模型（LLMs）与人类价值观和安全约束对齐具有挑战性，尤其是当有用性、真实性和避免伤害等目标相互冲突时。基于人类反馈的强化学习（RLHF）在引导模型方面取得了显著成功，但过程复杂且可能不稳定。最近的方法如直接偏好优化（DPO）简化了基于偏好的微调，但可能引入偏差或在某些目标上做出权衡。在本工作中，我们提出了一种基于组相对策略优化（GRPO）框架，并配备多标签奖励回归模型，以实现安全且对齐的语言生成。GRPO算法通过比较采样响应的组来优化策略，无需单独的价值评论器，从而提高了训练效率。我们训练了一个奖励模型来预测多个对齐分数（例如安全性、有用性等），并将其组合成单一的奖励信号。我们为在GRPO中使用这种学习到的多方面奖励提供了理论推导，并讨论了其优势和局限性。实验上，我们的方法在0.5B、7B和14B参数规模的模型上，改进了语言生成任务中评估的所有安全性和质量指标，展示了目标的稳健平衡。我们将GRPO与基于PPO的RLHF和DPO进行比较，强调GRPO以显著更低的计算成本和显式的多目标处理实现了对齐。我们将开源所有训练好的模型，网址为https://huggingface.co/hydroxai。
英文摘要：
Aligning large language models (LLMs) with human values and safety constraints is challenging, especially when objectives like helpfulness, truthfulness, and avoidance of harm conflict. Reinforcement Learning from Human Feedback (RLHF) has achieved notable success in steering models, but is complex and can be unstable. Recent approaches such as Direct Preference Optimization (DPO) simplify preference-based fine-tuning but may introduce bias or trade-off certain objectives~\cite{dpo}. In this work, we propose a Group Relative Policy Optimization (GRPO) framework with a multi-label reward regression model to achieve safe and aligned language generation. The GRPO algorithm optimizes a policy by comparing groups of sampled responses, eliminating the need for a separate value critic and improving training efficiency~\cite{grpo}. We train a reward model to predict multiple alignment scores (e.g., safety, helpfulness, etc.), which are combined into a single reward signal. We provide a theoretical derivation for using this learned multi-aspect reward within GRPO and discuss its advantages and limitations. Empirically, our approach improves all the safety and quality metrics evaluated in language generation tasks on model scales (0.5B, 7B, and 14B parameters), demonstrating a robust balance of objectives. We compare GRPO to PPO-based RLHF and DPO, highlighting that GRPO achieves alignment with significantly lower computational cost and explicit multi-objective handling. \textbf{We will open-source all trained models at https://huggingface.co/hydroxai.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
