arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.07635cs.LGcs.NE
收藏

神经元群局部学习

Local Learning with Neuron Groups

Concordia University(康考迪亚大学);Mila – Quebec AI Institute(米拉——魁北克人工智能研究所);Flatiron Institute(熨斗研究所)

Adeetya Patel, Michael Eickenberg, Eugene Belilovsky

摘要:

传统的深度网络训练方法为所有组件联合优化单一目标函数。这可能导致潜在并行化方面的各种低效率。局部学习是一种模型并行的方法,它消除了标准的端到端学习设置,并利用局部目标函数来允许深度网络中模型组件之间的并行学习。最近的研究表明,局部学习的变体可以有效地训练现代深度网络。然而,就可以分布多少计算而言,这些方法通常受到网络中层数的限制。在这项工作中,我们建议研究如何在将层或模块拆分为子组件的级别上应用本地学习,将宽度方向模块化的概念添加到与本地学习相关的现有深度方向模块化中。我们研究了本地学习惩罚,允许这样的模型被有效地训练。我们在CIFAR-10、CIFAR-100和Imagenet 32数据集上的实验表明,引入宽度级模块化可以带来优于现有基于局部学习的方法的计算优势,并为改进模型并行分布式训练提供新的机会。代码可从以下网址获得:https://github.com/adeetyapatel12/GN-DGL。

英文摘要:

Traditional deep network training methods optimize a monolithic objective function jointly for all the components. This can lead to various inefficiencies in terms of potential parallelization. Local learning is an approach to model-parallelism that removes the standard end-to-end learning setup and utilizes local objective functions to permit parallel learning amongst model components in a deep network. Recent works have demonstrated that variants of local learning can lead to efficient training of modern deep networks. However, in terms of how much computation can be distributed, these approaches are typically limited by the number of layers in a network. In this work we propose to study how local learning can be applied at the level of splitting layers or modules into sub-components, adding a notion of width-wise modularity to the existing depth-wise modularity associated with local learning. We investigate local-learning penalties that permit such models to be trained efficiently. Our experiments on the CIFAR-10, CIFAR-100, and Imagenet32 datasets demonstrate that introducing width-level modularity can lead to computational advantages over existing methods based on local learning and opens new opportunities for improved model-parallel distributed training. Code is available at: https://github.com/adeetyapatel12/GN-DGL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑