超越相关性分析的NLG评估指标：一份实证指标偏好清单
NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist
AI总结：
本文提出指标偏好清单框架，用于在文本摘要、对话回复生成和受控生成任务中检验自动评估指标对人类偏好的忠实度及系统级性能辨别能力，发现自动指标在部分任务中优于人类指导，且多维度人类对齐指标并不必然占优。
摘要：
在本研究中，我们分析了自然语言生成（NLG）的自动评估指标，具体包括任务无关指标和人类对齐指标。任务无关指标，如Perplexity、BLEU、BERTScore，具有成本效益高、对多样化NLG任务适应性强的特点，但它们与人类的相关性较弱。人类对齐指标（CTC、CtrlEval、UniEval）通过将理想的人类特质作为训练目标来提升相关性水平。然而，它们在辨别系统级性能和系统输出质量方面的有效性仍不明确。我们提出指标偏好清单作为一个框架，用于评估自动指标在三个NLG任务中的有效性：文本摘要、对话回复生成和受控生成。我们提出的框架提供以下功能：（i）验证自动指标是否忠实于人类偏好，无论其与人类的相关性水平如何；（ii）通过成对评估来考察NLG系统的优势与局限。我们表明，在文本摘要和受控生成任务中，自动指标在辨别系统级性能方面比人类提供更好的指导。我们还表明，多维度人类对齐指标（UniEval）并不必然优于单维度人类对齐指标（CTC、CtrlEval）和任务无关指标（BLEU、BERTScore），尤其是在受控生成任务中。
英文摘要：
In this study, we analyze automatic evaluation metrics for Natural Language Generation (NLG), specifically task-agnostic metrics and human-aligned metrics. Task-agnostic metrics, such as Perplexity, BLEU, BERTScore, are cost-effective and highly adaptable to diverse NLG tasks, yet they have a weak correlation with human. Human-aligned metrics (CTC, CtrlEval, UniEval) improves correlation level by incorporating desirable human-like qualities as training objective. However, their effectiveness at discerning system-level performance and quality of system outputs remain unclear. We present metric preference checklist as a framework to assess the effectiveness of automatic metrics in three NLG tasks: Text Summarization, Dialogue Response Generation, and Controlled Generation. Our proposed framework provides access: (i) for verifying whether automatic metrics are faithful to human preference, regardless of their correlation level to human; and (ii) for inspecting the strengths and limitations of NLG systems via pairwise evaluation. We show that automatic metrics provide a better guidance than human on discriminating system-level performance in Text Summarization and Controlled Generation tasks. We also show that multi-aspect human-aligned metric (UniEval) is not necessarily dominant over single-aspect human-aligned metrics (CTC, CtrlEval) and task-agnostic metrics (BLEU, BERTScore), particularly in Controlled Generation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
