arXiv 2306.056852023-12-24cs.CLcs.AI

用MT-Bench和Chatbot Arena评判LLM-as-a-Judge

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

UC Berkeley（加州大学伯克利分校）；UC San Diego（加州大学圣迭戈分校）；Carnegie Mellon University（卡内基梅隆大学）；Stanford（斯坦福大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica

备注：: NeurIPS 2023 Datasets and Benchmarks Track

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出使用强LLM作为裁判评估聊天助手，通过MT-bench和Chatbot Arena验证，发现GPT-4等裁判与人类偏好一致性超80%，提供可扩展且可解释的偏好近似方法。

摘要：

评估基于大语言模型（LLM）的聊天助手具有挑战性，因为其能力广泛，且现有基准在衡量人类偏好方面存在不足。为解决此问题，我们探索使用强大的LLM作为裁判来评估这些模型在更开放式问题上的表现。我们考察了LLM-as-a-judge的使用和局限性，包括位置偏差、冗长偏差和自我增强偏差，以及有限的推理能力，并提出缓解其中一些问题的解决方案。然后，我们通过引入两个基准来验证LLM裁判与人类偏好的一致性：MT-bench，一个多轮问题集；以及Chatbot Arena，一个众包对战平台。我们的结果显示，像GPT-4这样的强大LLM裁判能够很好地匹配受控和众包的人类偏好，达到超过80%的一致性，这与人类之间的一致性水平相同。因此，LLM-as-a-judge是一种可扩展且可解释的方式来近似人类偏好，而获取人类偏好通常代价高昂。此外，我们通过评估LLaMA和Vicuna的几种变体，展示了我们的基准与传统基准的互补性。MT-bench问题、3K专家投票和30K带人类偏好的对话可在https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge公开获取。

英文摘要：

Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。