arXiv 2012.007082020-12stat.MLcs.CLcs.LG

蒙特卡罗目标的互信息约束

Mutual Information Constraints for Monte-Carlo Objectives

DeepMind（深度思维）；University College London（伦敦大学学院）；Oxford University（牛津大学）

Gábor Melis, András György, Phil Blunsom

备注：: 32 pages, 29 figures

摘要：

作为变分自编码器训练的密度模型的一个常见故障模式是在不依赖于其潜变量的情况下对数据进行建模，从而使这些变量变得无用。文献中分别研究了两个影响因素，模型的欠规格和变分下限的松散性。我们编织这两股的研究在一起，特别是更严格的限制蒙特卡罗目标和约束的可观察变量和潜变量之间的互信息。将互信息估计为容易获得的变分后验$q(z|x)$和先验之间的平均库尔贝克-莱布勒散度，对于蒙特卡罗目标不起作用，因为$q(z|x)$不再是模型的真实后验$p(z|x)$的直接近似。因此，我们通过回收目标中使用的样本，构建了真实后验的库尔贝克-莱布勒散度的估计量，用它我们训练了连续和离散潜伏期模型，大大改善了率失真，没有后验崩溃。虽然减轻，建模数据和使用的潜在之间的权衡仍然存在，我们敦促评估推理方法在一系列的互信息值。

英文摘要：

A common failure mode of density models trained as variational autoencoders is to model the data without relying on their latent variables, rendering these variables useless. Two contributing factors, the underspecification of the model and the looseness of the variational lower bound, have been studied separately in the literature. We weave these two strands of research together, specifically the tighter bounds of Monte-Carlo objectives and constraints on the mutual information between the observable and the latent variables. Estimating the mutual information as the average Kullback-Leibler divergence between the easily available variational posterior $q(z|x)$ and the prior does not work with Monte-Carlo objectives because $q(z|x)$ is no longer a direct approximation to the model's true posterior $p(z|x)$. Hence, we construct estimators of the Kullback-Leibler divergence of the true posterior from the prior by recycling samples used in the objective, with which we train models of continuous and discrete latents at much improved rate-distortion and no posterior collapse. While alleviated, the tradeoff between modelling the data and using the latents still remains, and we urge for evaluating inference methods across a range of mutual information values.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文