arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1910.04859cs.CV
收藏

文本生成中的分布离散性检测

The Detection of Distributional Discrepancy for Text Generation

Nanjing University(南京大学);Southwest Jiaotong University(西南交通大学);Leshan Normal University(乐山师范学院)

Xingyuan Chen, Ping Cai, Peng Jin, Haokun Du, Hongjun Wang, Xingyu Dai, Jiajun Chen

摘要:

神经语言模型生成的文本不如真实文本。这意味着它们的分布是不同的。生成对抗网络(GAN)被用来缓解它。然而,一些研究人员认为,GAN变体根本不起作用。当同时考虑样本质量(如Bleu)和样本多样性(如self-Bleu)时,GAN变体甚至比调整良好的语言模型更差。但是,Bleu和self-Bleu不能精确地测量这种分布差异。事实上,如何度量真实文本和生成文本之间的分布差异仍然是一个悬而未决的问题。在本文中,我们从理论上提出了两个度量函数来衡量真实文本和生成文本之间的分布差异。此外,还提出了一种估算它们的方法。首先,我们用这两个函数对语言模型进行评估,发现两者的差异是巨大的。然后,我们尝试了几种方法来使用检测到的差异信号来改进发生器。然而,差距变得比以前更大了。在两种现有的语言GANs上进行实验,真实文本和生成文本之间的分布差异随着对抗性学习轮次的增加而增加。它证明了这两种语言GANs都失败了。

英文摘要:

The text generated by neural language models is not as good as the real text. This means that their distributions are different. Generative Adversarial Nets (GAN) are used to alleviate it. However, some researchers argue that GAN variants do not work at all. When both sample quality (such as Bleu) and sample diversity (such as self-Bleu) are taken into account, the GAN variants even are worse than a well-adjusted language model. But, Bleu and self-Bleu can not precisely measure this distributional discrepancy. In fact, how to measure the distributional discrepancy between real text and generated text is still an open problem. In this paper, we theoretically propose two metric functions to measure the distributional difference between real text and generated text. Besides that, a method is put forward to estimate them. First, we evaluate language model with these two functions and find the difference is huge. Then, we try several methods to use the detected discrepancy signal to improve the generator. However the difference becomes even bigger than before. Experimenting on two existing language GANs, the distributional discrepancy between real text and generated text increases with more adversarial learning rounds. It demonstrates both of these language GANs fail.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑