联合能量模型的稳定化训练及其实际应用
Stabilized training of joint energy-based models and their practical applications
摘要:
最近提出的基于能量的联合模型(JEM)将区分训练的分类器$p(y|x)$解释为能量模型,其也被训练为描述输入观测$p(x)$的分布的生成模型。JEM训练依赖于“正例”(即来自训练数据集的示例)以及“负例”,负例是来自通过随机梯度朗之万动力学(SGLD)生成的建模分布$p(x)$的样本。不幸的是,SGLD在标准JEM训练期间通常无法提供足够质量的负样本,这导致在计算JEM更新的梯度时,正样本和负样本的贡献非常不平衡。因此,标准的JEM训练非常不稳定,需要仔细调整超参数,并在训练开始发散时频繁重启。这使得很难将JEM应用于不同的神经网络架构、模态和任务。在这项工作中,我们提出了一个训练过程,通过平衡正面和负面示例的贡献来稳定基于SGLD的JEM训练(ST-JEM)。我们还建议在训练目标中添加一个额外的“正则化”项--输入观察$x$和输出标签$y$之间的MI--这鼓励JEM分类器对输出标签做出更确定的决定。我们证明了我们的方法在CIFAR 10和CIFAR 100任务上的有效性。我们还考虑语音信号中的音素分类的任务,为此,我们不能训练JEM没有建议的稳定。我们表明,一个令人信服的语音可以从训练模型生成。或者,可以通过使用几次SGLD迭代使受损语音更接近建模语音分布来对受损语音进行降噪。我们还提出并讨论了训练模型的其他应用。
英文摘要:
The recently proposed Joint Energy-based Model (JEM) interprets discriminatively trained classifier $p(y|x)$ as an energy model, which is also trained as a generative model describing the distribution of the input observations $p(x)$. The JEM training relies on "positive examples" (i.e. examples from the training data set) as well as on "negative examples", which are samples from the modeled distribution $p(x)$ generated by means of Stochastic Gradient Langevin Dynamics (SGLD). Unfortunately, SGLD often fails to deliver negative samples of sufficient quality during the standard JEM training, which causes a very unbalanced contribution from the positive and negative examples when calculating gradients for JEM updates. As a consequence, the standard JEM training is quite unstable requiring careful tuning of hyper-parameters and frequent restarts when the training starts diverging. This makes it difficult to apply JEM to different neural network architectures, modalities, and tasks. In this work, we propose a training procedure that stabilizes SGLD-based JEM training (ST-JEM) by balancing the contribution from the positive and negative examples. We also propose to add an additional "regularization" term to the training objective -- MI between the input observations $x$ and output labels $y$ -- which encourages the JEM classifier to make more certain decisions about output labels. We demonstrate the effectiveness of our approach on the CIFAR10 and CIFAR100 tasks. We also consider the task of classifying phonemes in a speech signal, for which we were not able to train JEM without the proposed stabilization. We show that a convincing speech can be generated from the trained model. Alternatively, corrupted speech can be de-noised by bringing it closer to the modeled speech distribution using a few SGLD iterations. We also propose and discuss additional applications of the trained model.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。