arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.11239cs.CL
收藏

LLM-as-a-Judge与奖励模型:它们能做什么和不能做什么

LLM-as-a-Judge & Reward Model: What They Can and Cannot Do

Yonsei University(延世大学);OnelineAI;Hankuk University of Foreign Studies(韩国外国语大学);Kangwon National University(江原国立大学);The University of Suwon(水原大学)

Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, Seunghyeok Hong

AI总结:

本文系统分析LLM-as-a-Judge与奖励模型在跨语言、事实性和复杂推理场景中的评估能力,揭示其英语能力迁移优势及错误检测、难题评估等局限,并开放数据集与代码。

摘要:

LLM-as-a-Judge和奖励模型被广泛用作多项选择题或人工标注者的替代方案,以评估大型语言模型(LLM)。它们在评估长文本回答方面表现出色,在排行榜评估中发挥关键作用,并作为通过强化学习对齐LLM的代理。然而,尽管它们广受欢迎,其在非英语提示、事实验证或高难度问题等多种情境下的有效性仍未得到探索。本文对自动化评估器进行了全面分析,报告了关于其行为的若干关键发现。首先,我们发现英语评估能力会显著影响特定语言的评估能力,且其影响往往超过语言熟练度本身,这使得在英语上训练的评估器能够轻松将技能迁移到其他语言。其次,我们发现了关键缺陷:LLM无法检测并惩罚错误,例如事实不准确、文化误述以及出现不受欢迎的语言。最后,我们发现最先进的评估器在处理英语或韩语的高难度提示时都存在困难,这凸显了它们在评估或生成复杂推理问题方面的局限。我们公开了所用的数据集和代码。

英文摘要:

LLM-as-a-Judge and reward models are widely used alternatives of multiple-choice questions or human annotators for large language model (LLM) evaluation. Their efficacy shines in evaluating long-form responses, serving a critical role as evaluators of leaderboards and as proxies to align LLMs via reinforcement learning. However, despite their popularity, their effectiveness in diverse contexts, such as non-English prompts, factual verification, or challenging questions, remains unexplored. In this paper, we conduct a comprehensive analysis of automated evaluators, reporting several key findings on their behavior. First, we discover that English evaluation capabilities significantly influence language-specific evaluation capabilities, often more than the language proficiency itself, enabling evaluators trained in English to easily transfer their skills to other languages. Second, we identify critical shortcomings, where LLMs fail to detect and penalize errors, such as factual inaccuracies, cultural misrepresentations, and the presence of unwanted language. Finally, we find that state-of-the-art evaluators struggle with challenging prompts, in either English or Korean, underscoring their limitations in assessing or generating complex reasoning questions. We release the dataset and codes used.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑