评估评估者：超越英语的摘要评估指标与大语言模型评判模型
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
AI总结：
该研究针对英语外语言开展摘要元评估，构建多语言数据集BASSE并测试自动摘要指标与LLM评判模型，发现专有评判LLM与人类评判相关性最高，开源评判LLM表现较差。
摘要：
自动文本摘要依赖自动评估，以通过自动指标和大语言模型评判模型（LLM-as-a-Judge）快速判定摘要模型的质量。然而，这些技术需要元评估，以确保其能准确捕捉人类评判。本文中，我们针对英语以外的语言开展该元评估，生成新的多语言摘要元评估数据集BASSE，包含对2040篇抽象摘要的人类评判，这些摘要由人工或5款大语言模型（LLMs）采用4种不同提示生成。针对每篇摘要，标注员采用5点李克特量表评估5项标准：连贯性、一致性、流畅性、相关性及5W1H。随后，我们对自动摘要指标和LLM-as-a-Judge模型进行基准测试。结果表明，当前专有评判LLM与人类评判的相关性最高，其次是特定标准的自动指标，而开源评判LLM表现不佳。
英文摘要：
Automatic text summarization relies on automatic evaluation to quickly determine the quality of summarization models via automatic metrics and LLM-as-a-Judge models. However, these techniques require meta-evaluation to ensure that they capture human judgments correctly. In this paper, we explore this meta-evaluation beyond English by generating a new multilingual summary meta-evaluation dataset (BASSE), which comprises human judgments on 2,040 abstractive summaries, generated either manually or by five Large Language Models (LLMs) with four different prompts. For each summary, annotators evaluate five criteria on a 5-point Likert scale: coherence, consistency, fluency, relevance, and 5W1H. We then benchmark automatic summarization metrics and LLM-as-a-Judge models. Our results show that currently proprietary judge LLMs have the highest correlation with human judgments, followed by criteria-specific automatic metrics, while open-sourced judge LLMs perform poorly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
