arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2012.14660cs.CL
收藏

语篇生成中重复问题的理论分析

A Theoretical Analysis of the Repetition Problem in Text Generation

The Chinese University of Hong Kong(香港中文大学);Tencent(腾讯)

Zihao Fu, Wai Lam, Anthony Man-Cho So, Bei Shi

摘要:

文本生成任务,包括翻译、摘要、语言模型等,近年来得到了快速的发展。尽管取得了显着的成就,重复的问题已被观察到几乎所有的文本生成模型破坏生成性能广泛。为了解决重复问题,已经提出了许多方法,但是没有现有的理论分析来说明为什么会发生这个问题以及如何解决这个问题。在本文中,我们提出了一个新的理论分析框架的重复问题。我们首先定义了平均重复概率(ARP)来定量描述重复问题。然后,我们进行了广泛的分析的马尔可夫生成模型,并得出了几个上界的平均重复概率与直观的理解。我们发现,大多数现有的方法基本上是显式或隐式最小化上界。基于我们的理论,我们表明,重复问题,不幸的是,我们的语言本身的特点造成的。一个主要原因是由于存在太多的词以高概率预测相同的词作为随后的词。因此,很容易回到这个词,形成重复,我们称之为高流入问题。此外,我们推导出一般生成模型的平均重复概率的浓度界。最后,基于理论上界,我们提出了一种新的再平衡编码方法来缓解高流入问题。实验结果表明,我们的理论框架是适用于一般的生成模型和我们提出的重新平衡编码方法,重复问题显着。本文的源代码可以从www.example.com获得。

英文摘要:

Text generation tasks, including translation, summarization, language models, and etc. see rapid growth during recent years. Despite the remarkable achievements, the repetition problem has been observed in nearly all text generation models undermining the generation performance extensively. To solve the repetition problem, many methods have been proposed, but there is no existing theoretical analysis to show why this problem happens and how it is resolved. In this paper, we propose a new framework for theoretical analysis for the repetition problem. We first define the Average Repetition Probability (ARP) to characterize the repetition problem quantitatively. Then, we conduct an extensive analysis of the Markov generation model and derive several upper bounds of the average repetition probability with intuitive understanding. We show that most of the existing methods are essentially minimizing the upper bounds explicitly or implicitly. Grounded on our theory, we show that the repetition problem is, unfortunately, caused by the traits of our language itself. One major reason is attributed to the fact that there exist too many words predicting the same word as the subsequent word with high probability. Consequently, it is easy to go back to that word and form repetitions and we dub it as the high inflow problem. Furthermore, we derive a concentration bound of the average repetition probability for a general generation model. Finally, based on the theoretical upper bounds, we propose a novel rebalanced encoding approach to alleviate the high inflow problem. The experimental results show that our theoretical framework is applicable in general generation models and our proposed rebalanced encoding approach alleviates the repetition problem significantly. The source code of this paper can be obtained from https://github.com/fuzihaofzh/repetition-problem-nlg.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑