迈向多参考时代——解决自然语言生成评估中的数据泄漏与参考多样性受限问题
Towards Multiple References Era -- Addressing Data Leakage and Limited Reference Diversity in NLG Evaluation
AI总结：
本文提出利用多参考文本提升BLEU等匹配型NLG评估指标与人类评估的一致性，在WMT Metrics上准确率超越单参考BLEU和BERTscore，并能有效缓解大语言模型的数据泄漏问题。
摘要：
基于N-gram匹配的评估指标，如BLEU和chrF，在各类自然语言生成（NLG）任务中被广泛使用。然而，近期研究表明，这些基于匹配的指标与人类评估之间的相关性较弱，尤其是与BLEURT等基于神经网络的指标相比。在本文中，我们推测基于匹配的指标的性能瓶颈可能源于参考文本的多样性有限。为解决这一问题，我们提出利用多个参考文本来增强这些指标与人类评估之间的一致性。在WMT Metrics基准测试中，我们观察到多参考F200spBLEU相比传统的单参考版本准确率提升了7.2%。值得注意的是，它还比基于神经网络的BERTscore准确率高出3.9%。此外，我们观察到，我们的多参考指标可以在很大程度上缓解大语言模型（LLMs）中的数据泄漏问题。我们在https://github.com/SefaZeng/LLM-Ref发布了代码和数据。
英文摘要：
N-gram matching-based evaluation metrics, such as BLEU and chrF, are widely utilized across a range of natural language generation (NLG) tasks. However, recent studies have revealed a weak correlation between these matching-based metrics and human evaluations, especially when compared with neural-based metrics like BLEURT. In this paper, we conjecture that the performance bottleneck in matching-based metrics may be caused by the limited diversity of references. To address this issue, we propose to utilize \textit{multiple references} to enhance the consistency between these metrics and human evaluations. Within the WMT Metrics benchmarks, we observe that the multi-references F200spBLEU surpasses the conventional single-reference one by an accuracy improvement of 7.2\%. Remarkably, it also exceeds the neural-based BERTscore by an accuracy enhancement of 3.9\%. Moreover, we observe that the data leakage issue in large language models (LLMs) can be mitigated to a large extent by our multi-reference metric. We release the code and data at \url{https://github.com/SefaZeng/LLM-Ref}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
