Think-J: 生成式大语言模型作为评判者的学习思考
Think-J: Learning to Think for Generative LLM-as-a-Judge
AI总结：
Think-J通过学习思考方式提升生成式LLM作为评判者的性能，利用强化学习优化判断轨迹，无需额外标注即可超越现有方法。
摘要：
LLM-as-a-Judge指的是对大型语言模型（LLMs）生成的响应进行自动建模以获取偏好，这对LLM评估和奖励建模具有重要意义。尽管生成式LLMs在各种任务上取得了显著进展，但其作为LLM-Judge的性能仍未能达到预期。在本工作中，我们提出了Think-J，通过学习如何思考来提升生成式LLM-as-a-Judge。我们首先利用少量经过精心挑选的数据来开发具有初始判断思考能力的模型。随后，我们基于强化学习（RL）优化判断思考轨迹。我们提出了两种判断思考优化方法，分别基于离线和在线RL。离线方法需要训练一个批评者模型来构建正例和负例以进行学习。在线方法则定义基于规则的奖励作为优化的反馈。实验结果表明，我们的方法可以显著提升生成式LLM-Judge的评估能力，超越了生成式和分类器基于的LLM-Judge，且无需额外的人类注释。
英文摘要：
LLM-as-a-Judge refers to the automatic modeling of preferences for responses generated by Large Language Models (LLMs), which is of significant importance for both LLM evaluation and reward modeling. Although generative LLMs have made substantial progress in various tasks, their performance as LLM-Judge still falls short of expectations. In this work, we propose Think-J, which improves generative LLM-as-a-Judge by learning how to think. We first utilized a small amount of curated data to develop the model with initial judgment thinking capabilities. Subsequently, we optimize the judgment thinking traces based on reinforcement learning (RL). We propose two methods for judgment thinking optimization, based on offline and online RL, respectively. The offline method requires training a critic model to construct positive and negative examples for learning. The online method defines rule-based reward as feedback for optimization. Experimental results showed that our approach can significantly enhance the evaluation capability of generative LLM-Judge, surpassing both generative and classifier-based LLM-Judge without requiring extra human annotations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
