我们提出了一种用于局部自监督表示学习的新型深度学习方法,该方法不需要标签,也不需要端到端反向传播,而是利用数据中的自然顺序。受生物神经网络似乎在没有反向传播全局错误信号的情况下进行学习的观察的启发,我们将深度神经网络拆分为梯度隔离模块的堆栈。每个模块都经过训练,以使用Oord et al. [2018]的InfoNCE界限最大限度地保留其输入的信息。尽管有这种贪婪的训练,我们证明了每个模块都改进了其前任的输出,并且顶部模块创建的表示在音频和视觉领域的下游分类任务中产生了非常有竞争力的结果。该提案能够异步优化模块,允许在未标记的数据集上对非常深的神经网络进行大规模分布式训练。
英文摘要:
We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal, we split a deep neural network into a stack of gradient-isolated modules. Each module is trained to maximally preserve the information of its inputs using the InfoNCE bound from Oord et al. [2018]. Despite this greedy training, we demonstrate that each module improves upon the output of its predecessor, and that the representations created by the top module yield highly competitive results on downstream classification tasks in the audio and visual domain. The proposal enables optimizing modules asynchronously, allowing large-scale distributed training of very deep neural networks on unlabelled datasets.
Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supply
College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院);ZJU-UIUC Institute, Zhejiang University(浙江大学-伊利诺伊大学厄巴纳-香槟分校研究院)
Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation
King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);University of Oxford(牛津大学)