检查表比奖励模型更能对齐语言模型
Checklists Are Better Than Reward Models For Aligning Language Models
AI总结：
本文提出基于检查表反馈的强化学习方法，通过灵活的指令特定准则提升语言模型的指令遵循能力，在多个基准测试中均取得性能提升。
摘要：
语言模型必须适应以理解和遵循用户指令。强化学习被广泛用于促进这一点--通常使用固定的准则，如“有用性”和“有害性”。在我们的工作中，我们提出使用灵活的、针对指令的准则作为扩大强化学习在促使遵循指令方面影响的手段。我们提出“基于检查表反馈的强化学习”（RLCF）。从指令中提取检查表，并评估响应满足每个项目的情况--使用AI评判者和专门的验证程序进行评估，然后将这些分数结合起来计算强化学习的奖励。我们比较了RLCF与其他对齐方法在强指令遵循模型（Qwen2.5-7B-Instruct）上的表现，五个广泛研究的基准测试中，RLCF是唯一一个在每个基准测试上都提高性能的方法，包括在FollowBench上硬满足率的4点提升，在InFoBench上的6点增加，在Arena-Hard上的赢率3点上升。这些结果确立了检查表反馈作为改进语言模型支持表达多种需求的查询的关键工具。
英文摘要：
Language models must be adapted to understand and follow user instructions. Reinforcement learning is widely used to facilitate this -- typically using fixed criteria such as "helpfulness" and "harmfulness". In our work, we instead propose using flexible, instruction-specific criteria as a means of broadening the impact that reinforcement learning can have in eliciting instruction following. We propose "Reinforcement Learning from Checklist Feedback" (RLCF). From instructions, we extract checklists and evaluate how well responses satisfy each item - using both AI judges and specialized verifier programs - then combine these scores to compute rewards for RL. We compare RLCF with other alignment methods applied to a strong instruction following model (Qwen2.5-7B-Instruct) on five widely-studied benchmarks -- RLCF is the only method to improve performance on every benchmark, including a 4-point boost in hard satisfaction rate on FollowBench, a 6-point increase on InFoBench, and a 3-point rise in win rate on Arena-Hard. These results establish checklist feedback as a key tool for improving language models' support of queries that express a multitude of needs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
