arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.15067cs.CL
收藏

并非所有指标都有过错:通过多样化参考改进自然语言生成评估

Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References

Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);The Chinese University of Hong Kong(香港中文大学);AIWaves Inc.(AIWaves公司);Microsoft Research Asia, China(微软亚洲研究院);Microsoft(微软公司)

Tianyi Tang, Hongyuan Lu, Yuchen Eleanor Jiang, Haoyang Huang, Dongdong Zhang, Wayne Xin Zhao, Tom Kocmi, Furu Wei

AI总结:

本文提出Div-Ref方法,利用大语言模型将单一参考多样化,丰富评估基准的参考数量,显著提升自然语言生成自动评估与人类评估的相关性,并鼓励未来基准包含更多参考。

摘要:

大多数关于自然语言生成(NLG)的研究依赖于每个样本仅有有限参考的评估基准,这可能导致与人类判断的相关性较差。其根本原因在于,一个语义含义实际上可以用不同的形式表达,而使用单一或少量参考的评估可能无法准确反映模型假设的质量。为解决这一问题,本文提出了一种简单有效的方法,名为Div-Ref,通过丰富参考数量来增强现有评估基准。我们利用大型语言模型(LLMs)将单一参考的表达多样化,生成多个高质量的参考,以尽可能覆盖参考句子的语义空间。我们进行了全面的实验,实证表明,多样化参考的表达可以显著增强自动评估与人类评估之间的相关性。这一想法与近期基于LLM的评估兼容,后者同样可以从引入多个参考中获益。我们强烈鼓励未来的生成基准包含更多参考,即使这些参考是由LLMs生成的,这是一劳永逸的。我们在https://github.com/RUCAIBox/Div-Ref发布了所有代码和数据,以促进研究。

英文摘要:

Most research about natural language generation (NLG) relies on evaluation benchmarks with limited references for a sample, which may result in poor correlations with human judgements. The underlying reason is that one semantic meaning can actually be expressed in different forms, and the evaluation with a single or few references may not accurately reflect the quality of the model's hypotheses. To address this issue, this paper presents a simple and effective method, named Div-Ref, to enhance existing evaluation benchmarks by enriching the number of references. We leverage large language models (LLMs) to diversify the expression of a single reference into multiple high-quality ones to cover the semantic space of the reference sentence as much as possible. We conduct comprehensive experiments to empirically demonstrate that diversifying the expression of reference can significantly enhance the correlation between automatic evaluation and human evaluation. This idea is compatible with recent LLM-based evaluation which can similarly derive advantages from incorporating multiple references. We strongly encourage future generation benchmarks to include more references, even if they are generated by LLMs, which is once for all. We release all the code and data at https://github.com/RUCAIBox/Div-Ref to facilitate research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑