AutoRule：推理链提取的基于规则的奖励改善偏好学习
AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning
AI总结：
AutoRule自动从偏好反馈中提取规则作为辅助奖励，结合推理链与验证器，显著提升Llama-3-8B在AlpacaEval2.0和MT-Bench上的性能，并减少奖励黑客行为。
摘要：
基于规则的奖励为改进从人类反馈中进行强化学习（RLHF）提供了一种有前景的策略，但当前方法往往依赖于人工规则工程。我们提出了AutoRule，一种从偏好反馈中提取规则并将其形式化为基于规则的奖励的全自动方法。AutoRule的提取过程分为三个阶段：它利用推理模型来解释用户偏好，从这些解释的推理链中识别候选规则，并将它们综合成一个统一的规则集。利用最终确定的规则集，我们采用语言模型验证器来计算每个输出满足规则的比例，并将该指标作为辅助奖励，在策略优化过程中与学习到的奖励模型一起使用。使用AutoRule训练Llama-3-8B模型，在AlpacaEval2.0上的长度控制胜率相对提高了28.6%，在保留的MT-Bench子集上的第二轮表现相对提高了6.1%，与使用相同学习奖励模型但无基于规则的辅助奖励的GRPO基线相比。我们的分析证实，提取的规则与数据集偏好具有良好的一致性。我们发现，与学习到的奖励模型相比，AutoRule在运行两个回合时表现出减少的奖励黑客行为。最后，我们的案例研究表明，提取的规则捕捉了不同数据集中所重视的独特品质。提取的规则在附录中提供，代码已在https://github.com/cxcscmu/AutoRule开源。
英文摘要：
Rule-based rewards offer a promising strategy for improving reinforcement learning from human feedback (RLHF), but current approaches often rely on manual rule engineering. We present AutoRule, a fully automated method for extracting rules from preference feedback and formulating them into rule-based rewards. AutoRule extraction operates in three stages: it leverages a reasoning model to interpret user preferences, identifies candidate rules from the reasoning chain of these interpretations, and synthesizes them into a unified rule set. Leveraging the finalized rule set, we employ language-model verifiers to compute the fraction of rules satisfied by each output, using this metric as an auxiliary reward alongside the learned reward model during policy optimization. Training a Llama-3-8B model with AutoRule results in a 28.6\% relative improvement in length-controlled win rate on AlpacaEval2.0, and a 6.1\% relative gain in second-turn performance on a held-out MT-Bench subset, compared to a GRPO baseline trained with the same learned reward model but without the rule-based auxiliary reward. Our analysis confirms that the extracted rules exhibit good agreement with dataset preference. We find that AutoRule demonstrates reduced reward hacking compared to a learned reward model when run over two episodes. Finally, our case study suggests that the extracted rules capture unique qualities valued in different datasets. The extracted rules are provided in the appendix, and the code is open-sourced at https://github.com/cxcscmu/AutoRule.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
