HRLAIF：开放域从AI反馈中进行强化学习的帮助性与无害性改进
HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback
AI总结：
针对基础RLAIF在开放域中导致响应满意度下降的问题，提出HRLAIF方法，通过增强AI注释准确性和AI红队测试，提升模型帮助性与无害性，使满意度较RL前提高2.08%。
摘要：
从AI反馈中进行强化学习（RLAIF）相比从人类反馈中进行强化学习（RLHF）具有注释周期更短和成本更低的优势，使其在大语言模型（LLM）训练的快速策略迭代期间非常高效。在RLAIF训练中使用ChatGPT作为标注器为开放域提示提供反馈，我们观察到人类评估者对模型响应的偏好胜率有所增加，但评估者的满意度却有所下降。分析表明，满意度的下降主要源于一些响应变得不够有帮助，尤其是在正确性和真实性方面，这凸显了基础RLAIF的实际局限性。在本文中，我们提出了混合从AI反馈中进行强化学习（HRLAIF）。该方法增强了AI对响应注释的准确性，使模型在训练过程中的帮助性更加稳健。此外，它采用AI进行红队测试，进一步提高了模型的无害性。人类评估结果表明，HRLAIF继承了RLAIF以低成本增强人类对结果偏好的能力，同时也提高了响应的满意度。与强化学习（RL）之前的策略模型相比，满意度提高了2.08%，有效解决了基础RLAIF后满意度下降4.58%的问题。
英文摘要：
Reinforcement Learning from AI Feedback (RLAIF) has the advantages of shorter annotation cycles and lower costs over Reinforcement Learning from Human Feedback (RLHF), making it highly efficient during the rapid strategy iteration periods of large language model (LLM) training. Using ChatGPT as a labeler to provide feedback on open-domain prompts in RLAIF training, we observe an increase in human evaluators' preference win ratio for model responses, but a decrease in evaluators' satisfaction rate. Analysis suggests that the decrease in satisfaction rate is mainly due to some responses becoming less helpful, particularly in terms of correctness and truthfulness, highlighting practical limitations of basic RLAIF. In this paper, we propose Hybrid Reinforcement Learning from AI Feedback (HRLAIF). This method enhances the accuracy of AI annotations for responses, making the model's helpfulness more robust in training process. Additionally, it employs AI for Red Teaming, further improving the model's harmlessness. Human evaluation results show that HRLAIF inherits the ability of RLAIF to enhance human preference for outcomes at a low cost while also improving the satisfaction rate of responses. Compared to the policy model before Reinforcement Learning (RL), it achieves an increase of 2.08\% in satisfaction rate, effectively addressing the issue of a decrease of 4.58\% in satisfaction rate after basic RLAIF.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
