arXiv 2411.165942025-09-29cs.AIcs.CL

从生成到判断：LLM-as-a-judge 的机遇与挑战

From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge

Arizona State University（亚利桑那州立大学）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；Northwestern University（西北大学）；University of California, Berkeley（加州大学伯克利分校）；Emory University（埃默里大学）

Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu

备注：: EMNLP 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文综述 LLM-as-a-judge 范式，从输入输出定义、判断对象与方法及基准测试三维构建分类体系，并指出关键挑战与未来方向。

摘要：

评估与评价长期以来一直是人工智能（AI）和自然语言处理（NLP）中的关键挑战。传统方法通常基于匹配或小模型，在开放式和动态场景中往往存在不足。大型语言模型（LLM）的最新进展激发了“LLM-as-a-judge”范式，即利用 LLM 在各类机器学习评估场景中执行评分、排序或选择。本文对基于 LLM 的判断与评估进行了全面综述，提供深入概览以回顾这一不断发展的领域。我们首先从输入和输出两个角度给出定义。随后，我们介绍一种系统分类法，沿三个维度探讨 LLM-as-a-judge：判断什么、如何判断，以及如何进行基准测试。最后，我们还强调了这一新兴领域的关键挑战和有前景的未来方向。有关 LLM-as-a-judge 的更多资源可访问网站：https://llm-as-a-judge.github.io 和 https://github.com/llm-as-a-judge/Awesome-LLM-as-a-judge。

英文摘要：

Assessment and evaluation have long been critical challenges in artificial intelligence (AI) and natural language processing (NLP). Traditional methods, usually matching-based or small model-based, often fall short in open-ended and dynamic scenarios. Recent advancements in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm, where LLMs are leveraged to perform scoring, ranking, or selection for various machine learning evaluation scenarios. This paper presents a comprehensive survey of LLM-based judgment and assessment, offering an in-depth overview to review this evolving field. We first provide the definition from both input and output perspectives. Then we introduce a systematic taxonomy to explore LLM-as-a-judge along three dimensions: what to judge, how to judge, and how to benchmark. Finally, we also highlight key challenges and promising future directions for this emerging area. More resources on LLM-as-a-judge are on the website: https://llm-as-a-judge.github.io and https://github.com/llm-as-a-judge/Awesome-LLM-as-a-judge.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。