Recent approximations to backpropagation (BP) have mitigated many of BP's computational inefficiencies and incompatibilities with biology, but important limitations still remain. Moreover, the approximations significantly decrease accuracy in benchmarks, suggesting that an entirely different approach may be more fruitful. Here, grounded on recent theory for Hebbian learning in soft winner-take-all networks, we present multilayer SoftHebb, i.e. an algorithm that trains deep neural networks, without any feedback, target, or error signals. As a result, it achieves efficiency by avoiding weight transport, non-local plasticity, time-locking of layer updates, iterative equilibria, and (self-) supervisory or other feedback signals -- which were necessary in other approaches. Its increased efficiency and biological compatibility do not trade off accuracy compared to state-of-the-art bio-plausible learning, but rather improve it. With up to five hidden layers and an added linear classifier, accuracies on MNIST, CIFAR-10, STL-10, and ImageNet, respectively reach 99.4%, 80.3%, 76.2%, and 27.3%. In conclusion, SoftHebb shows with a radically different approach from BP that Deep Learning over few layers may be plausible in the brain and increases the accuracy of bio-plausible machine learning. Code is available at https://github.com/NeuromorphicComputing/SoftHebb.
Deep Neural Networks using a Single Neuron: Folded-in-Time Architecture using Feedback-Modulated Delay Loops
Institute of Mathematics, Technische Universität Berlin(柏林工业大学数学研究所);Department of Mathematics, Humboldt-Universität zu Berlin(柏林洪堡大学数学系);Institute of Computer Science, University of Tartu(塔尔图大学计算机科学研究所);Instituto de Física Interdisciplinar y Sistemas Complejos, IFISC (UIB-CSIC)(跨学科物理与复杂系统研究所(巴利阿里群岛大学-西班牙高等科研理事会))
正在进行的研究已经确定了生物网络和深度人工神经网络中神经表征之间的相似性。这使得人们对开发用于训练人工网络的反向传播学习算法与大脑中的突触可塑性规则之间的类比重新产生了兴趣。这些努力受到反向传播的生物学上难以置信的特征的挑战,其中之一是对对称的前向和后向突触权重的依赖。已经提出了许多不依赖于权重对称的方法,但到目前为止,这些方法未能扩展到深度卷积网络和复杂数据。我们确定的主要障碍,这种算法的可扩展性,并介绍了几种技术来减轻它们。我们证明,修改后的反馈对齐方法,强制执行较弱形式的权重对称性,一个需要协议的重量符号,但不幅度,可以实现与反向传播的性能竞争。我们的研究结果补充了Bartunov et al.(2018)和Xiao et al.(2018 b)的研究结果,并表明促进前馈和反馈权重对齐的机制对于深度网络中的学习至关重要。