自生成批评提升语言模型的奖励建模
Self-Generated Critiques Boost Reward Modeling for Language Models
AI总结：
针对奖励模型难以纳入自然语言批评的问题，提出Critic-RM框架，通过自生成并过滤高质量批评、联合微调奖励预测与批评生成，提升奖励建模准确率3.7%-7.3%，还能改善推理步骤。
摘要：
奖励建模对于将大型语言模型（LLMs）与人类偏好对齐至关重要，尤其是在基于人类反馈的强化学习（RLHF）中。然而，当前的奖励模型主要生成标量分数，难以以自然语言格式纳入批评。我们假设同时预测批评和标量奖励将提高奖励建模能力。基于此，我们提出Critic-RM框架，无需额外监督即可利用自生成批评改进奖励模型。Critic-RM采用两阶段流程：生成并过滤高质量批评，随后在奖励预测和批评生成任务上进行联合微调。跨基准实验表明，与标准奖励模型和LLM评判器相比，Critic-RM将奖励建模准确率提升3.7%-7.3%，展现出强大的性能和数据效率。额外研究进一步验证了生成的批评在纠正有缺陷的推理步骤方面的有效性，推理准确率提升2.5%-3.2%。
英文摘要：
Reward modeling is crucial for aligning large language models (LLMs) with human preferences, especially in reinforcement learning from human feedback (RLHF). However, current reward models mainly produce scalar scores and struggle to incorporate critiques in a natural language format. We hypothesize that predicting both critiques and the scalar reward would improve reward modeling ability. Motivated by this, we propose Critic-RM, a framework that improves reward models using self-generated critiques without extra supervision. Critic-RM employs a two-stage process: generating and filtering high-quality critiques, followed by joint fine-tuning on reward prediction and critique generation. Experiments across benchmarks show that Critic-RM improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges, demonstrating strong performance and data efficiency. Additional studies further validate the effectiveness of generated critiques in rectifying flawed reasoning steps with 2.5%-3.2% gains in improving reasoning accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
