分析与评估 NLG 元评估中的相关性度量
Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluation
AI总结：
本文分析了 NLG 元评估中 12 种相关性度量，利用六个数据集和 32 个指标的真实数据，提出判别力、排序一致性和分数粒度敏感性三个评估视角，发现全局分组结合 Pearson 相关系数表现最佳。
摘要：
NLG 自动评估指标与人工评估之间的相关性通常被视为评估指标能力的关键标准。然而，不同的分组方法和相关系数导致了元评估中使用的多种相关性度量。在特定评估场景中，先前的工作通常直接遵循常规的度量设置，但这些度量之间的特征和差异未得到充分关注。因此，本文利用来自六个广泛使用的 NLG 评估数据集和 32 个评估指标的大量真实数据，分析了 12 种常见的相关性度量，揭示了不同度量确实会影响元评估结果。此外，我们提出了反映元评估能力的三个视角：判别力、排序一致性和对分数粒度的敏感性。我们发现，使用全局分组和 Pearson 相关系数的度量在判别力和排序一致性方面表现最佳。此外，使用系统级分组或 Kendall 相关性的度量对分数粒度的敏感性最低。
英文摘要：
The correlation between NLG automatic evaluation metrics and human evaluation is often regarded as a critical criterion for assessing the capability of an evaluation metric. However, different grouping methods and correlation coefficients result in various types of correlation measures used in meta-evaluation. In specific evaluation scenarios, prior work often directly follows conventional measure settings, but the characteristics and differences between these measures have not gotten sufficient attention. Therefore, this paper analyzes 12 common correlation measures using a large amount of real-world data from six widely-used NLG evaluation datasets and 32 evaluation metrics, revealing that different measures indeed impact the meta-evaluation results. Furthermore, we propose three perspectives that reflect the capability of meta-evaluation: discriminative power, ranking consistency, and sensitivity to score granularity. We find that the measure using global grouping and Pearson correlation coefficient exhibits the best performance in both discriminative power and ranking consistency. Besides, the measures using system-level grouping or Kendall correlation are the least sensitive to score granularity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
