arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1901.08164cs.LGstat.ML
收藏

CNN的解耦贪婪学习

Decoupled Greedy Learning of CNNs

MILA(蒙特利尔学习算法研究所);University of California, Berkeley(加州大学伯克利分校);CNRS(法国国家科学研究中心)

Eugene Belilovsky, Michael Eickenberg, Edouard Oyallon

摘要:

一个经常被引用的反向传播神经网络训练的低效率是更新锁定问题:每一层都必须等待信号在更新之前通过整个网络传播。已经提出了几种可以缓解这个问题的替代方案。在这种情况下,我们考虑一个更简单,但更有效的替代品,使用最小的反馈,我们称之为解耦贪婪学习(DGL)。它基于联合训练目标的贪婪松弛,最近在大规模图像分类的卷积神经网络(CNN)的背景下被证明是有效的。我们考虑对这个目标进行优化,使我们能够解耦层训练,允许网络中的层或模块在层中进行潜在的线性并行化训练。使用一个重放缓冲区,我们表明这种方法可以扩展到异步设置,模块可以操作可能很大的通信延迟。我们从理论上和经验上表明,这种方法收敛。然后,我们实证发现,它可以导致更好的推广比顺序贪婪优化。我们在CIFAR-10数据集和大规模ImageNet数据集上证明了DGL对替代方法的有效性。

英文摘要:

A commonly cited inefficiency of neural network training by back-propagation is the update locking problem: each layer must wait for the signal to propagate through the full network before updating. Several alternatives that can alleviate this issue have been proposed. In this context, we consider a simpler, but more effective, substitute that uses minimal feedback, which we call Decoupled Greedy Learning (DGL). It is based on a greedy relaxation of the joint training objective, recently shown to be effective in the context of Convolutional Neural Networks (CNNs) on large-scale image classification. We consider an optimization of this objective that permits us to decouple the layer training, allowing for layers or modules in networks to be trained with a potentially linear parallelization in layers. With the use of a replay buffer we show this approach can be extended to asynchronous settings, where modules can operate with possibly large communication delays. We show theoretically and empirically that this approach converges. Then, we empirically find that it can lead to better generalization than sequential greedy optimization. We demonstrate the effectiveness of DGL against alternative approaches on the CIFAR-10 dataset and on the large-scale ImageNet dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑