MM-Eval：一个用于LLM作为评判者和奖励模型的多语言元评估基准
MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
AI总结：
本文提出了多语言元评估基准MM-Eval，用于检验评估器LLMs在多语言输出上的可靠性与公平性，发现现有评估器在非英语及低资源语言上表现不佳且不一致，且该基准与Best-of-N排名具有更强相关性。
摘要：
随着大型语言模型（LLMs）现在能够生成除英语之外其他语言的流畅且连贯的内容，精确评估这些非英语输出变得至关重要。然而，在评估多语言LLMs的输出时，先前的工作通常采用擅长评估英语输出的基于LLM的评估器，而没有仔细检查这些评估器是否也能有效评估非英语文本。此外，用于测试评估器LLMs的现有基准（被称为“元评估基准”）主要以英语为中心。为了弥补这一差距并检查评估器LLMs能否可靠地评估多语言LLMs的输出，我们引入了MM-Eval，这是一个多语言元评估基准，包含覆盖18种语言的五个核心子集，以及一个跨越122种语言的语言一致性子集。MM-Eval的一个核心属性是，它并非仅仅翻译现有的英语元评估基准，而是在设计时考虑了多语言特定的挑战。此外，与仅关注成对数据排名准确性的现有元评估基准不同，MM-Eval还评估了在广泛语言范围内绝对分数值的一致性和公平性。我们的结果表明，在英语语境中表现出色的现有评估器LLMs在评估非英语输出时有很大的改进空间。此外，我们发现评估器在评估较低资源语言时是不公平且不一致的。最后，我们通过测量MM-Eval与Best-of-N排名的相关性来验证MM-Eval，发现与其他元评估基准相比存在显著更强的相关性。我们公开发布了我们的基准和代码。
英文摘要：
As Large Language Models (LLMs) are now capable of producing fluent and coherent content in languages other than English, it is not imperative to precisely evaluate these non-English outputs. However, when assessing the outputs from mutlilingual LLMs, prior works often employed LLM based evaluators that excel at assessing English outputs, without a thorough examination of whether these evaluators could effectively assess non-English text as well. Moreover, existing benchmarks to test evaluator LLMs (referred to as "meta-evaluation benchmarks") are mostly English-centric. To bridge this gap and examine whether evaluator LLMs can reliably assess the outputs of multilingual LLMs, we introduce MM-Eval, a multilingual meta-evaluation benchmark comprising five core subsets covering 18 languages and a Language Consistency subset spanning 122 languages. A core attribute of MM-Eval is that, instead of merely translating existing English meta-evaluation benchmarks, it is designed with multilingual-specific challenges in mind. Additionally, unlike existing meta-evaluation benchmarks that focus solely on ranking accuracy over pairwise data, MM-Eval also evaluates the consistency and fairness of absolute score values across a wide range of languages. Our results show that existing evaluator LLMs that excel in English contexts have considerable room for improvement when assessing non-English outputs. Furthermore, we find that evaluators are unfair and inconsistent when evaluating lower-resourced languages. Finally, we validate MM-Eval by measuring its correlation with Best-of-N rankings, finding a significantly stronger correlation compared to other meta-evaluation benchmarks. We publicly release our benchmark and code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
