基于PLM的困惑度对文本质量评价的不可靠性
Perplexity from PLM Is Unreliable for Evaluating Text Quality
摘要：
近年来，大量的研究工作利用困惑度PPL来评价生成文本的质量。他们认为，如果PPL值较小，则要评估的文本的质量（即流畅性）更好。然而，我们发现PPL裁判员是不合格的，它不能公平地评估生成的文本，原因如下：（i）短文本的PPL大于长文本，这违背了常识;（ii）重复的文本跨度可能会损害PPL的性能;（iii）标点符号可能会严重影响PPL的性能。实验表明，PPL是不可靠的评价给定的文本的质量。最后，我们讨论了使用语言模型评估文本质量的关键问题。
英文摘要：
Recently, amounts of works utilize perplexity~(PPL) to evaluate the quality of the generated text. They suppose that if the value of PPL is smaller, the quality(i.e. fluency) of the text to be evaluated is better. However, we find that the PPL referee is unqualified and it cannot evaluate the generated text fairly for the following reasons: (i) The PPL of short text is larger than long text, which goes against common sense, (ii) The repeated text span could damage the performance of PPL, and (iii) The punctuation marks could affect the performance of PPL heavily. Experiments show that the PPL is unreliable for evaluating the quality of given text. Last, we discuss the key problems with evaluating text quality using language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
