arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2208.02531cs.CL
收藏

InitialGAN:一种完全随机初始化的语言 GAN

InitialGAN: A Language GAN with Completely Random Initialization

The Hong Kong Polytechnic University(香港理工大学)

Da Ren, Qing Li

AI总结:

针对语言 GAN 依赖预训练的问题,本文提出结合 dropout 采样与全归一化 LSTM、完全随机初始化的 InitialGAN,并用最小覆盖率评估,首次实现无预训练语言 GAN 超越 MLE。

摘要:

通过最大似然估计(MLE)训练的文本生成模型存在众所周知的暴露偏差问题,而生成对抗网络(GAN)已被证明在解决该问题方面具有潜力。现有的语言 GAN 采用 REINFORCE 等估计器或连续松弛方法来建模词概率。此类估计器的固有局限导致当前模型依赖预训练技术(MLE 预训练或预训练嵌入)。然而,不受这些局限影响的表示建模方法,由于此前尝试中的性能较差而很少得到探索。我们的分析表明,无效的采样方法和不健康的梯度是造成这种不理想性能的主要原因。在本研究中,我们提出两种技术来解决这些问题:dropout 采样和全归一化 LSTM。基于这两项技术,我们提出 InitialGAN,其参数完全随机初始化。此外,我们引入一种新的评估指标——最小覆盖率(Least Coverage Rate),以更好地评估生成样本的质量。实验结果表明,InitialGAN 的表现优于 MLE 以及其他参与比较的模型。据我们所知,这是语言 GAN 首次能够在不使用任何预训练技术的情况下超越 MLE。

英文摘要:

Text generative models trained via Maximum Likelihood Estimation (MLE) suffer from the notorious exposure bias problem, and Generative Adversarial Networks (GANs) are shown to have potential to tackle this problem. Existing language GANs adopt estimators like REINFORCE or continuous relaxations to model word probabilities. The inherent limitations of such estimators lead current models to rely on pre-training techniques (MLE pre-training or pre-trained embeddings). Representation modeling methods which are free from those limitations, however, are seldomly explored because of their poor performance in previous attempts. Our analyses reveal that invalid sampling methods and unhealthy gradients are the main contributors to such unsatisfactory performance. In this work, we present two techniques to tackle these problems: dropout sampling and fully normalized LSTM. Based on these two techniques, we propose InitialGAN whose parameters are randomly initialized in full. Besides, we introduce a new evaluation metric, Least Coverage Rate, to better evaluate the quality of generated samples. The experimental results demonstrate that InitialGAN outperforms both MLE and other compared models. To the best of our knowledge, it is the first time a language GAN can outperform MLE without using any pre-training techniques.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑