过滤式直接偏好优化
Filtered Direct Preference Optimization
AI总结：
针对DPO对偏好数据文本质量高度敏感的问题，提出fDPO，利用训练好的奖励模型在训练中过滤低质量样本，从而提升最终模型性能。
摘要：
基于人类反馈的强化学习（RLHF）在使语言模型与人类偏好保持一致方面发挥着关键作用。尽管数据集质量的重要性已得到普遍认可，但据我们所知，在RLHF框架内对其影响进行的明确研究仍然有限。本文聚焦直接偏好优化（DPO）这一日益被采用的无奖励模型RLHF方法，以解决偏好数据集中的文本质量问题。我们证实，文本质量对采用DPO优化的模型性能的影响，显著大于对基于奖励模型的RLHF所优化模型的影响。基于这一新认识，我们提出了DPO的一种扩展，称为过滤式直接偏好优化（fDPO）。fDPO使用经过训练的奖励模型，在DPO训练期间监控偏好数据集中文本的质量。通过与正在优化的模型所生成的文本进行比较，质量较低的样本会被丢弃，从而得到更准确的数据集。实验结果表明，fDPO提升了最终模型的性能。我们的代码可在https://github.com/CyberAgentAILab/filtered-dpo获取。
英文摘要：
Reinforcement learning from human feedback (RLHF) plays a crucial role in aligning language models with human preferences. While the significance of dataset quality is generally recognized, explicit investigations into its impact within the RLHF framework, to our knowledge, have been limited. This paper addresses the issue of text quality within the preference dataset by focusing on direct preference optimization (DPO), an increasingly adopted reward-model-free RLHF method. We confirm that text quality significantly influences the performance of models optimized with DPO more than those optimized with reward-model-based RLHF. Building on this new insight, we propose an extension of DPO, termed filtered direct preference optimization (fDPO). fDPO uses a trained reward model to monitor the quality of texts within the preference dataset during DPO training. Samples of lower quality are discarded based on comparisons with texts generated by the model being optimized, resulting in a more accurate dataset. Experimental results demonstrate that fDPO enhances the final model performance. Our code is available at https://github.com/CyberAgentAILab/filtered-dpo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
