RRHF:无需繁琐操作即可通过排序响应实现语言模型与人类反馈对齐
RRHF: Rank Responses to Align Language Models with Human Feedback without tears
AI总结:
针对RLHF中PPO训练复杂、难扩展的问题,提出RRHF范式,通过排序损失对齐模型响应概率与人类偏好,仅需1-2个模型且超参数调优简单,在Helpful and Harmless数据集上达到与PPO相当的对齐性能。
摘要:
基于人类反馈的强化学习(Reinforcement Learning from Human Feedback, RLHF)推动大语言模型与人类偏好对齐,显著提升了人机交互质量。InstructGPT通过多个阶段实现RLHF,包括监督微调(Supervised Fine-Tuning, SFT)、奖励模型训练以及近端策略优化(Proximal Policy Optimization, PPO)。但PPO对超参数敏感,标准实现需要多个模型,训练难度大且难以扩展到更大参数规模。\n对此,我们提出一种名为RRHF的新型学习范式,它通过条件概率的对数对不同来源的采样响应进行评分,并通过排序损失学习将这些概率与人类偏好对齐。RRHF可以利用来自各种来源的采样响应,包括自身的模型响应、其他大语言模型的响应以及人类专家的响应,来学习对它们进行排序。RRHF在微调过程中仅需1到2个模型,无需复杂的超参数调优,就能高效、鲁棒地使语言模型与人类偏好对齐。此外,RRHF可被视为SFT和奖励模型训练的扩展,在代码实现、模型数量和超参数方面都比PPO更简单。\n我们在Helpful and Harmless数据集上评估了RRHF,结果显示,从奖励模型得分和人工标注来看,RRHF的对齐性能与PPO相当。大量实验表明,RRHF的性能与采样质量高度相关,这说明RRHF是一种best-of-n学习器。代码可在https://github.com/GanjinZero/RRHF获取。
英文摘要:
Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). However, PPO is sensitive to hyperparameters and requires multiple models in its standard implementation, making it hard to train and scale up to larger parameter counts. In contrast, we propose a novel learning paradigm called RRHF, which scores sampled responses from different sources via a logarithm of conditional probabilities and learns to align these probabilities with human preferences through ranking loss. RRHF can leverage sampled responses from various sources including the model responses from itself, other large language model responses, and human expert responses to learn to rank them. RRHF only needs 1 to 2 models during tuning and can efficiently align language models with human preferences robustly without complex hyperparameter tuning. Additionally, RRHF can be considered an extension of SFT and reward model training while being simpler than PPO in terms of coding, model counts, and hyperparameters. We evaluate RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment performance with PPO by reward model score and human labeling. Extensive experiments show that the performance of RRHF is highly related to sampling quality which suggests RRHF is a best-of-n learner. Codes available at https://github.com/GanjinZero/RRHF.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。