arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2204.09890cs.CL
收藏

文本生成无参考评价中的虚假相关

Spurious Correlations in Reference-Free Evaluation of Text Generation

Stanford University(斯坦福大学);Columbia University(哥伦比亚大学)

Esin Durmus, Faisal Ladhak, Tatsunori Hashimoto

摘要:

基于模型的、无参考的评价指标已被提出作为评价自然语言生成(NLG)系统的快速且成本有效的方法。尽管最近的结果很有希望,我们发现证据表明,无参考评估指标的总结和对话生成可能依赖于虚假的相关性与措施,如字重叠,困惑,和长度。我们进一步观察到,对于文本摘要,这些指标在对当前最先进的抽象摘要系统进行排名时具有高出错率。我们证明,这些错误可以减轻明确设计的评价指标,以避免虚假的功能,在无参考评价。

英文摘要:

Model-based, reference-free evaluation metrics have been proposed as a fast and cost-effective approach to evaluate Natural Language Generation (NLG) systems. Despite promising recent results, we find evidence that reference-free evaluation metrics of summarization and dialog generation may be relying on spurious correlations with measures such as word overlap, perplexity, and length. We further observe that for text summarization, these metrics have high error rates when ranking current state-of-the-art abstractive summarization systems. We demonstrate that these errors can be mitigated by explicitly designing evaluation metrics to avoid spurious features in reference-free evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑