arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.24322cs.LGstat.ML
收藏

NoProp:无需完整反向传播或完整前向传播的神经网络训练方法

NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation

Department of Statistics University of Oxford(统计系牛津大学)

Qinyu Li, Yee Whye Teh, Razvan Pascanu

AI总结:

本文提出 NoProp 训练方法,不依赖全局前向或反向传播,而是让每个模块独立学习对带噪目标进行局部去噪,并在多个图像分类基准上验证了其可行性与计算效率。

摘要:

深度学习的经典学习方法要求在每个模块通过将误差信号从输出端反向传播至每个可学习参数来计算梯度项。鉴于神经网络的分层堆叠结构,其中每个模块都建立在下方模块的表示之上,这种方法会产生层级化表示。更抽象的特征位于模型的顶层模块,而底层模块的特征则预期较不抽象。与此相反,我们引入了一种名为 NoProp 的新学习方法,它不依赖整个网络的前向或反向传播。NoProp 从扩散模型和流匹配方法中汲取灵感,其中每个模块仅使用局部目标和模块内部的反向传播,独立学习对带噪目标进行去噪。我们相信这项工作朝着引入一类不学习层级化表示的新学习方法迈出了第一步——至少不是通常意义上的层级化表示。NoProp 需要预先将每个模块的表示固定为目标的一个带噪版本,学习一个局部去噪过程,随后可在推理时加以利用。我们在 MNIST、CIFAR-10 和 CIFAR-100 图像分类基准上证明了该方法的有效性。结果表明,NoProp 是一种可行的学习算法,易于使用且计算高效。通过脱离需要反向传播全局误差信号的传统学习范式,NoProp 改变了网络内部的信用分配方式,从而支持更高效的分布式学习,并可能影响学习过程的其他特性。

英文摘要:

The canonical deep learning approach for learning requires computing a gradient term at each block by back-propagating the error signal from the output towards each learnable parameter. Given the stacked structure of neural networks, where each block builds on the representation of the block below, this approach leads to hierarchical representations. More abstract features live on the top blocks of the model, while features on lower blocks are expected to be less abstract. In contrast to this, we introduce a new learning method named NoProp, which does not rely on either forward or backwards propagation across the entire network. Instead, NoProp takes inspiration from diffusion and flow matching methods, where each block independently learns to denoise a noisy target using only local targets and back-propagation within the block. We believe this work takes a first step towards introducing a new family of learning methods that does not learn hierarchical representations -- at least not in the usual sense. NoProp needs to fix the representation at each block beforehand to a noised version of the target, learning a local denoising process that can then be exploited at inference. We demonstrate the effectiveness of our method on MNIST, CIFAR-10, and CIFAR-100 image classification benchmarks. Our results show that NoProp is a viable learning algorithm, is easy to use and computationally efficient. By departing from the traditional learning paradigm which requires back-propagating a global error signal, NoProp alters how credit assignment is done within the network, enabling more efficient distributed learning as well as potentially impacting other characteristics of the learning process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑