arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1905.11786cs.LGcs.AIstat.ML
收藏

结束端到端:独立自主的表征学习

Putting An End to End-to-End: Gradient-Isolated Learning of Representations

AMLab;University of Amsterdam(阿姆斯特丹大学)

Sindy Löwe, Peter O'Connor, Bastiaan S. Veeling

摘要:

我们提出了一种用于局部自监督表示学习的新型深度学习方法,该方法不需要标签,也不需要端到端反向传播,而是利用数据中的自然顺序。受生物神经网络似乎在没有反向传播全局错误信号的情况下进行学习的观察的启发,我们将深度神经网络拆分为梯度隔离模块的堆栈。每个模块都经过训练,以使用Oord et al. [2018]的InfoNCE界限最大限度地保留其输入的信息。尽管有这种贪婪的训练,我们证明了每个模块都改进了其前任的输出,并且顶部模块创建的表示在音频和视觉领域的下游分类任务中产生了非常有竞争力的结果。该提案能够异步优化模块,允许在未标记的数据集上对非常深的神经网络进行大规模分布式训练。

英文摘要:

We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal, we split a deep neural network into a stack of gradient-isolated modules. Each module is trained to maximally preserve the information of its inputs using the InfoNCE bound from Oord et al. [2018]. Despite this greedy training, we demonstrate that each module improves upon the output of its predecessor, and that the representations created by the top module yield highly competitive results on downstream classification tasks in the audio and visual domain. The proposal enables optimizing modules asynchronously, allowing large-scale distributed training of very deep neural networks on unlabelled datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑