用于语言生成的GANs的精确评估
On Accurate Evaluation of GANs for Language Generation
摘要:
生成对抗网络(GANs)是一种很有前途的语言生成方法。最新的作品引入了新的GAN模型用于语言生成,使用基于n-gram的指标进行评估,并且只报告最佳运行的单个分数。在本文中,我们认为,这往往歪曲了真实情况,并没有告诉完整的故事,因为GAN模型可以是非常敏感的随机初始化和最佳超参数选择的小偏差。特别是,我们证明了以前使用的BLEU分数是不敏感的生成的文本的语义恶化,并提出替代指标,更好地捕捉生成的样本的质量和多样性。我们还进行了一组实验,比较了一些GAN模型的文本与传统的语言模型(LM),并发现所考虑的模型都没有令人信服地表现得比LM更好。
英文摘要:
Generative Adversarial Networks (GANs) are a promising approach to language generation. The latest works introducing novel GAN models for language generation use n-gram based metrics for evaluation and only report single scores of the best run. In this paper, we argue that this often misrepresents the true picture and does not tell the full story, as GAN models can be extremely sensitive to the random initialization and small deviations from the best hyperparameter choice. In particular, we demonstrate that the previously used BLEU score is not sensitive to semantic deterioration of generated texts and propose alternative metrics that better capture the quality and diversity of the generated samples. We also conduct a set of experiments comparing a number of GAN models for text with a conventional Language Model (LM) and find that neither of the considered models performs convincingly better than the LM.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。