arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2006.03824cs.NE
收藏

通过减小梯度估计偏差的方法研究标度平衡向深部ConvNets的传播

Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias

Axel Laborieux, Maxence Ernoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier, Damien Querlioz

摘要:

平衡传播(EP)是一种生物启发的收敛RNN算法,具有强有力的理论保证的局部学习规则。在信用分配阶段期间,神经网络的参数更新已经在数学上被示出为当网络被无限小地推向其目标时,接近通过时间的反向传播(BPTT)提供的梯度。然而,在实践中,用EP提供的梯度估计训练网络并不能扩展到比MNIST更难的视觉任务。在这项工作中,我们证明了EP的梯度估计中的偏差,这是使用有限轻推所固有的,是造成这种现象的原因,并且取消它可以通过EP训练深度ConvNets。我们表明,这种偏见可以大大减少使用对称轻推(一个积极的轻推和一个负)。我们还将以前的EP方程推广到交叉熵损失的情况下(通过反对平方误差)。由于这些进步,我们能够实现CIFAR-10的测试误差为11.7%的EP,这接近BPTT实现的测试误差,并提供了一个重大的改进,相对于标准的EP方法,同号轻推,给出86%的测试误差。我们也将这些技术应用于训练具有不对称前向和后向连接的架构,得到13.2%的测试误差。这些结果突出了EP作为一种令人信服的生物学上合理的方法来计算深度神经网络中的误差梯度。

英文摘要:

Equilibrium Propagation (EP) is a biologically-inspired algorithm for convergent RNNs with a local learning rule that comes with strong theoretical guarantees. The parameter updates of the neural network during the credit assignment phase have been shown mathematically to approach the gradients provided by Backpropagation Through Time (BPTT) when the network is infinitesimally nudged toward its target. In practice, however, training a network with the gradient estimates provided by EP does not scale to visual tasks harder than MNIST. In this work, we show that a bias in the gradient estimate of EP, inherent in the use of finite nudging, is responsible for this phenomenon and that cancelling it allows training deep ConvNets by EP. We show that this bias can be greatly reduced by using symmetric nudging (a positive nudging and a negative one). We also generalize previous EP equations to the case of cross-entropy loss (by opposition to squared error). As a result of these advances, we are able to achieve a test error of 11.7% on CIFAR-10 by EP, which approaches the one achieved by BPTT and provides a major improvement with respect to the standard EP approach with same-sign nudging that gives 86% test error. We also apply these techniques to train an architecture with asymmetric forward and backward connections, yielding a 13.2% test error. These results highlight EP as a compelling biologically-plausible approach to compute error gradients in deep neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑