RAFT：面向生成式基础模型对齐的奖励排序微调
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
AI总结：
本文提出RAFT框架，利用奖励模型对生成样本进行排序筛选，仅保留高质量样本进行微调，从而以更稳定高效的方式实现生成式基础模型与人类偏好对齐，并在大语言模型和扩散模型上验证了其有效性。
摘要：
生成式基础模型容易受到大规模无监督训练数据中可能产生的隐性偏见的影响。此类偏见可能导致次优样本、结果偏差和不公平现象，并可能带来严重后果。因此，使这些模型与人类伦理和偏好对齐，是确保其在现实世界应用中负责任且有效部署的关键步骤。先前研究主要采用基于人类反馈的强化学习（RLHF）来解决这一问题，即利用以人类反馈为依据的奖励模型引导生成模型进行强化学习微调。然而，强化学习算法通常存在的低效性和不稳定性，往往对成功对齐构成重大障碍，因此需要开发一种更稳健、更简化的方法。为此，我们提出了一种新框架——奖励排序微调（RAFT），旨在有效对齐生成模型。我们的方法利用奖励模型和足够数量的样本，筛选出高质量样本，丢弃表现出不良行为的样本，随后在这些过滤后的样本上进行微调以增强模型。我们的研究表明，在大语言模型和扩散模型中，RAFT 均能有效提升模型在奖励学习以及其他自动化指标上的表现。
英文摘要：
Generative foundation models are susceptible to implicit biases that can arise from extensive unsupervised training data. Such biases can produce suboptimal samples, skewed outcomes, and unfairness, with potentially serious consequences. Consequently, aligning these models with human ethics and preferences is an essential step toward ensuring their responsible and effective deployment in real-world applications. Prior research has primarily employed Reinforcement Learning from Human Feedback (RLHF) to address this problem, where generative models are fine-tuned with RL algorithms guided by a human-feedback-informed reward model. However, the inefficiencies and instabilities associated with RL algorithms frequently present substantial obstacles to the successful alignment, necessitating the development of a more robust and streamlined approach. To this end, we introduce a new framework, Reward rAnked FineTuning (RAFT), designed to align generative models effectively. Utilizing a reward model and a sufficient number of samples, our approach selects the high-quality samples, discarding those that exhibit undesired behavior, and subsequently enhancing the model by fine-tuning on these filtered samples. Our studies show that RAFT can effectively improve the model performance in both reward learning and other automated metrics in both large language models and diffusion models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
