全变差距离下的语言生成模型裁剪
Tailoring Language Generation Models under Total Variation Distance
摘要：
神经语言生成的标准范例采用最大似然估计（MLE）作为优化方法。从分布的角度来看，MLE实际上最小化了真实数据分布和模型分布之间的Kullback-Leibler散度（KLD）。然而，这种方法迫使模型将非零（有时很大）概率质量分配给所有训练样本，而不管它们的质量如何。此外，在试图覆盖数据分布中的低概率区域时，该模型系统地高估了损坏的文本序列的概率，我们推测这是自回归解码期间文本退化的主要原因之一。为了解决这个问题，我们利用总变差距离（TVD）的鲁棒性离群值，并开发实用的界限，将其应用到语言生成。然后，我们引入Tailr目标，平衡估计TVD的权衡。直观地说，TaiLr降低了具有低模型概率的真实数据样本的权重，并具有可调的惩罚强度。实验结果表明，我们的方法在不牺牲多样性的情况下消除了退化序列的高估，并在广泛的文本生成任务上提高了生成质量。
英文摘要：
The standard paradigm of neural language generation adopts maximum likelihood estimation (MLE) as the optimizing method. From a distributional view, MLE in fact minimizes the Kullback-Leibler divergence (KLD) between the distribution of the real data and that of the model. However, this approach forces the model to distribute non-zero (sometimes large) probability mass to all training samples regardless of their quality. Moreover, in the attempt to cover the low-probability regions in the data distribution, the model systematically overestimates the probability of corrupted text sequences, which we conjecture is one of the main reasons for text degeneration during autoregressive decoding. To remedy this problem, we leverage the total variation distance (TVD) with its robustness to outliers, and develop practical bounds to apply it to language generation. Then, we introduce the TaiLr objective that balances the tradeoff of estimating TVD. Intuitively, TaiLr downweights real data samples that have low model probabilities with tunable penalization intensity. Experimental results show that our method alleviates the overestimation of degenerated sequences without sacrificing diversity and improves generation quality on a wide range of text generation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
