arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.18099cs.AIcs.CL
收藏

学习规划与推理:以Thinking-LLM-as-a-Judge进行评估

Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge

FAIR at Meta(Meta 的 FAIR)

Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, Tianlu Wang

AI总结:

提出EvalPlanner,一种用于Thinking-LLM-as-a-Judge的偏好优化算法,通过先规划后执行再判断的自我训练循环,在RewardBench上以93.9分达到生成式奖励模型新最先进性能。

摘要:

LLM-as-a-Judge模型生成思维链(CoT)序列,旨在捕捉响应最终评估所依据的逐步推理过程。然而,由于缺乏用于评估的人工标注思维链,有效推理轨迹所需的组成部分和结构仍未得到充分研究。因此,以往的方法通常(1)将推理轨迹限制为手工设计的组件,如标准列表、参考答案或验证问题,以及(2)将其结构设计为规划与评估推理交织在一起。在本工作中,我们提出EvalPlanner,一种用于Thinking-LLM-as-a-Judge的偏好优化算法,该算法首先生成无约束的评估计划,然后执行该计划,最后给出最终判断。在自我训练循环中,EvalPlanner迭代优化基于合成构建的评估计划和执行,从而产生更好的最终裁决。我们的方法在RewardBench上取得了生成式奖励模型的新最先进性能(得分为93.9),尽管训练所用的偏好对数量更少且为合成生成。在RM-Bench、JudgeBench和FollowBenchEval等其他基准上的额外实验进一步凸显了规划和推理对于构建稳健的LLM-as-a-Judge推理模型的实用性。

英文摘要:

LLM-as-a-Judge models generate chain-of-thought (CoT) sequences intended to capture the step-bystep reasoning process that underlies the final evaluation of a response. However, due to the lack of human annotated CoTs for evaluation, the required components and structure of effective reasoning traces remain understudied. Consequently, previous approaches often (1) constrain reasoning traces to hand-designed components, such as a list of criteria, reference answers, or verification questions and (2) structure them such that planning is intertwined with the reasoning for evaluation. In this work, we propose EvalPlanner, a preference optimization algorithm for Thinking-LLM-as-a-Judge that first generates an unconstrained evaluation plan, followed by its execution, and then the final judgment. In a self-training loop, EvalPlanner iteratively optimizes over synthetically constructed evaluation plans and executions, leading to better final verdicts. Our method achieves a new state-of-the-art performance for generative reward models on RewardBench (with a score of 93.9), despite being trained on fewer amount of, and synthetically generated, preference pairs. Additional experiments on other benchmarks like RM-Bench, JudgeBench, and FollowBenchEval further highlight the utility of both planning and reasoning for building robust LLM-as-a-Judge reasoning models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑