REPA：用于评估文本生成与判断能力的俄语错误类型标注
REPA: Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilities
AI总结：
本文提出俄语错误类型标注数据集REPA，评估LLM-as-a-judge框架在俄语场景的表现，发现其俄语细粒度评估能力弱于英语，但人类与LLM的偏好排名存在部分一致性，具备改进空间。
摘要：
大语言模型（LLM）的最新进展催生了将LLM用作评委的新范式，即由一个LLM评估并打分另一个LLM的输出，这种方式通常与人类偏好高度相关。然而，LLM-as-a-judge（大语言模型作为评委）的研究主要集中在英语领域。本文通过引入俄语错误类型标注数据集REPA，在俄语语境下评估该框架，该数据集包含1000条用户查询和2000条LLM生成的回复。人类标注员对每对回复进行标注，从十种特定错误类型维度表达偏好，并选出整体偏好结果。我们基于人类偏好，采用三种评分体系对六款生成式LLM在不同错误类型上的表现进行排名。我们还在零样本和少样本设置下，使用八款LLM评委评估回复质量。本文阐述了对评委表现以及位置偏差、长度偏差的分析结果。研究发现，LLM评委在俄语和英语中的表现存在显著差距。不过，基于人类偏好和LLM偏好的排名呈现出部分一致性，这表明尽管当前LLM评委在俄语细粒度评估上存在困难，但仍有提升潜力。
英文摘要：
Recent advances in large language models (LLMs) have introduced the novel paradigm of using LLMs as judges, where an LLM evaluates and scores the outputs of another LLM, which often correlates highly with human preferences. However, the use of LLM-as-a-judge has been primarily studied in English. In this paper, we evaluate this framework in Russian by introducing the Russian Error tyPes Annotation dataset (REPA), a dataset of 1k user queries and 2k LLM-generated responses. Human annotators labeled each response pair expressing their preferences across ten specific error types, as well as selecting an overall preference. We rank six generative LLMs across the error types using three rating systems based on human preferences. We also evaluate responses using eight LLM judges in zero-shot and few-shot settings. We describe the results of analyzing the judges and position and length biases. Our findings reveal a notable gap between LLM judge performance in Russian and English. However, rankings based on human and LLM preferences show partial alignment, suggesting that while current LLM judges struggle with fine-grained evaluation in Russian, there is potential for improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
