JudgeLRM：作为评判者的大型推理模型
JudgeLRM: Large Reasoning Models as a Judge
AI总结：
针对LLM作为评估器在复杂推理任务上的不足，提出基于强化学习的JudgeLRM系列模型，通过评判导向的奖励机制激活推理能力，在多项基准上超越SFT基线及GPT-4、DeepSeek-R1等先进模型。
摘要：
大型语言模型（LLMs）越来越多地被用作评估器，为人类标注提供了可扩展的替代方案。然而，现有的监督微调（SFT）方法在需要复杂推理的领域中往往表现不佳。评判本质上是一项推理密集型任务：除了表面层面的打分，它还需要验证证据、识别错误并证明决策的合理性。通过对评估任务的分析，我们发现SFT的性能提升与需要推理的样本比例之间存在负相关，揭示了SFT在此类场景中的局限性。为了解决这一问题，我们引入了JudgeLRM，这是一系列面向评判的LLM，使用强化学习（RL）进行训练，并通过评判导向、结果驱动的奖励来激活推理能力。JudgeLRM在相同规模下持续优于SFT微调的基线，以及其他RL和SFT变体，甚至超越了最先进的推理模型：值得注意的是，JudgeLRM-3B/4B超越了GPT-4，而JudgeLRM-7B/8B/14B在F1分数上比DeepSeek-R1高出2%以上，在推理繁重的任务上表现尤为突出。我们的发现强调了RL在解锁推理对齐的LLM评判者方面的价值。
英文摘要：
Large Language Models (LLMs) are increasingly adopted as evaluators, offering a scalable alternative to human annotation. However, existing supervised fine-tuning (SFT) approaches often fall short in domains that demand complex reasoning. Judgment is inherently reasoning-intensive: beyond surface-level scoring, it requires verifying evidence, identifying errors, and justifying decisions. Through the analysis of evaluation tasks, we find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios. To address this, we introduce JudgeLRM, a family of judgment-oriented LLMs, trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards to activate reasoning capabilities. JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models: notably, JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1 score, with particularly strong gains on reasoning-heavy tasks. Our findings underscore the value of RL in unlocking reasoning-aligned LLM judges.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
