在 NLG 系统的评估中参考文本是否必要？何时以及何处需要？
Is Reference Necessary in the Evaluation of NLG Systems? When and Where?
AI总结：
本研究全面评估了无参考文本指标在8个数据集和8个评估模型上的表现，发现其与人类判断相关性更高，但有效性受任务和候选文本质量影响，建议在新任务中谨慎应用。
摘要：
大多数用于评估 NLG 系统的自动指标都是基于参考文本的。然而，收集人工标注的挑战导致在众多应用场景中缺乏可靠的参考文本。尽管无参考文本指标近期取得了进展，但人们尚未充分理解何时以及在何处可以将其作为基于参考文本指标的替代方案。在本研究中，通过采用多种分析方法，我们全面评估了这两类指标在广泛 NLG 任务上的表现，涵盖 8 个数据集和 8 个评估模型。基于扎实的实验，结果表明，无参考文本指标与人类判断表现出更高的相关性，并且对语言质量的缺陷具有更大的敏感性。然而，它们的有效性因任务而异，并受到候选文本质量的影响。因此，在将无参考文本指标应用于新任务之前，评估其性能非常重要，特别是当输入采用不常见形式或答案空间高度可变时。我们的研究能够为自动指标的恰当应用以及指标选择对评估性能的影响提供见解。
英文摘要：
The majority of automatic metrics for evaluating NLG systems are reference-based. However, the challenge of collecting human annotation results in a lack of reliable references in numerous application scenarios. Despite recent advancements in reference-free metrics, it has not been well understood when and where they can be used as an alternative to reference-based metrics. In this study, by employing diverse analytical approaches, we comprehensively assess the performance of both metrics across a wide range of NLG tasks, encompassing eight datasets and eight evaluation models. Based on solid experiments, the results show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. However, their effectiveness varies across tasks and is influenced by the quality of candidate texts. Therefore, it's important to assess the performance of reference-free metrics before applying them to a new task, especially when inputs are in uncommon form or when the answer space is highly variable. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
