对比发散和持续对比发散中方差的随机梯度估计
Stochastic Gradient Estimate Variance in Contrastive Divergence and Persistent Contrastive Divergence
摘要：
对比发散（CD）和持续对比发散（PCD）是训练受限玻尔兹曼机权值的常用方法。然而，这两种方法都使用近似方法从模型分布中进行采样。作为一个副作用，这些近似产生显着不同的偏差和方差的随机梯度估计的个别数据点。众所周知，CD产生有偏梯度估计。然而，在本文中，我们的经验表明，CD具有较低的随机梯度估计方差比精确采样，而随后的PCD估计的平均值具有较高的方差比精确采样。结果给出了一个解释的发现，CD可以使用更小的小批量或更高的学习率比PCD。
英文摘要：
Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) are popular methods for training the weights of Restricted Boltzmann Machines. However, both methods use an approximate method for sampling from the model distribution. As a side effect, these approximations yield significantly different biases and variances for stochastic gradient estimates of individual data points. It is well known that CD yields a biased gradient estimate. In this paper we however show empirically that CD has a lower stochastic gradient estimate variance than exact sampling, while the mean of subsequent PCD estimates has a higher variance than exact sampling. The results give one explanation to the finding that CD can be used with smaller minibatches or higher learning rates than PCD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
