通过在线策略自评判对齐大型语言模型
Aligning Large Language Models by On-Policy Self-Judgment
AI总结：
提出SELF-JUDGE框架，通过JSFT训练单一模型同时作为策略和评判者，实现在线策略对齐，无需额外奖励模型，实验证明其优于基线。
摘要：
现有的大型语言模型与人类偏好对齐方法面临权衡，需要在策略学习时使用单独的奖励模型（RM）。本文提出了一种新颖的对齐框架SELF-JUDGE，该框架（1）进行在线策略学习，（2）参数高效，因为它不需要额外的奖励模型来评估在线策略学习的样本。为此，我们提出了评判增强的监督微调（JSFT）来训练一个单一模型同时充当策略和评判者。具体来说，我们将成对评判任务（从一对响应中选择更好的响应）视为指令遵循任务的特例。由此产生的模型能够评判来自当前策略（由其自身初始化）的实时响应的偏好。实验结果表明SELF-JUDGE的有效性，在偏好基准上优于基线。我们还表明，拒绝采样本身可以进一步提高性能，而无需额外的评估器。
英文摘要：
Existing approaches for aligning large language models with human preferences face a trade-off that requires a separate reward model (RM) for on-policy learning. In this paper, we present a novel alignment framework, SELF-JUDGE that (1) does on-policy learning and 2) is parameter efficient, as it does not require an additional RM for evaluating the samples for on-policy learning. To this end, we propose Judge-augmented Supervised Fine-Tuning (JSFT) to train a single model to act as both a policy and a judge. Specifically, we view the pairwise judgment task, choosing the better response from a response pair, as a special case of the instruction-following task. The resulting model can judge preferences of on-the-fly responses from current policy initialized from itself. Experimental results show the efficacy of SELF-JUDGE, outperforming baselines in preference benchmarks. We also show that the rejecting sampling by itself can improve performance further without an additional evaluator.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
