arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.12563cs.CL
收藏

生成文本无参考评价的局限性

On the Limitations of Reference-Free Evaluations of Generated Text

Google Research(谷歌研究院);University of Pennsylvania(宾夕法尼亚大学)

Daniel Deutsch, Rotem Dror, Dan Roth

摘要:

有显着的兴趣,在开发的评估指标,准确地估计生成的文本的质量,而没有一个人写的参考文本的帮助下,这可能是耗时和昂贵的收集或完全不可用的在线应用程序。然而,在这项工作中,我们证明了这些无参考指标在评估生成的文本的能力方面存在固有的偏见和局限性,我们认为它们不应该用于衡量机器翻译或摘要等任务的进展。我们展示了无参考指标如何等同于使用一个生成模型来评估另一个生成模型,这有几个限制:(1)指标可以在测试时进行优化,以找到近似的最佳输出,(2)它们固有地偏向于与自己更相似的模型,(3)它们可能偏向于更高质量的输出,包括人类编写的输出。因此,我们建议使用无参考指标作为分析和理解模型行为的诊断工具,而不是测量模型执行任务的程度,其目标是尽可能获得高分。

英文摘要:

There is significant interest in developing evaluation metrics which accurately estimate the quality of generated text without the aid of a human-written reference text, which can be time consuming and expensive to collect or entirely unavailable in online applications. However, in this work, we demonstrate that these reference-free metrics are inherently biased and limited in their ability to evaluate generated text, and we argue that they should not be used to measure progress on tasks like machine translation or summarization. We show how reference-free metrics are equivalent to using one generation model to evaluate another, which has several limitations: (1) the metrics can be optimized at test time to find the approximate best-possible output, (2) they are inherently biased toward models which are more similar to their own, and (3) they can be biased against higher-quality outputs, including those written by humans. Therefore, we recommend that reference-free metrics should be used as diagnostic tools for analyzing and understanding model behavior instead of measures of how well models perform a task, in which the goal is to achieve as high of a score as possible.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑