Traditional deep network training methods optimize a monolithic objective function jointly for all the components. This can lead to various inefficiencies in terms of potential parallelization. Local learning is an approach to model-parallelism that removes the standard end-to-end learning setup and utilizes local objective functions to permit parallel learning amongst model components in a deep network. Recent works have demonstrated that variants of local learning can lead to efficient training of modern deep networks. However, in terms of how much computation can be distributed, these approaches are typically limited by the number of layers in a network. In this work we propose to study how local learning can be applied at the level of splitting layers or modules into sub-components, adding a notion of width-wise modularity to the existing depth-wise modularity associated with local learning. We investigate local-learning penalties that permit such models to be trained efficiently. Our experiments on the CIFAR-10, CIFAR-100, and Imagenet32 datasets demonstrate that introducing width-level modularity can lead to computational advantages over existing methods based on local learning and opens new opportunities for improved model-parallel distributed training. Code is available at: https://github.com/adeetyapatel12/GN-DGL.
A New Artificial Neuron Proposal with Trainable Simultaneous Local and Global Activation Function
Universidade Federal Rural de Pernambuco(伯南布哥联邦农村大学);Harvard University(哈佛大学)
The activation function plays a fundamental role in the artificial neural network learning process. However, there is no obvious choice or procedure to determine the best activation function , which depends on the problem. This study proposes a new artificial neuron, named global-local neuron, with a trainable activation function composed of two components, a global and a local. The global component term used here is relative to a mathematical function to describe a general feature present in all problem domain. The local component is a function that can represent a localized behavior, like a …
Quantifying Local Specialization in Deep Neural Networks
University of California Berkeley(加州大学伯克利分校);Boston University(波士顿大学);Center for Human-Compatible AI (CHAI)(人类兼容人工智能中心);MIT Computer Science and Artificial Intelligence Laboratory (CSAIL)(麻省理工学院计算机科学与人工智能实验室)
Computational analysis of laminar structure of the human cortex based on local neuron features
Croatian Institute for Brain Research(克罗地亚脑研究所);University of Zagreb, School of Medicine(萨格勒布大学医学院);Ruđer Bošković Institute(鲁杰尔·博什科维奇研究所);Faculty of Electrical Engineering and Computing(电气工程与计算学院);University of Zagreb(萨格勒布大学)
What should a neuron aim for? Designing local objective functions based on information theory
Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen(物理学院、复杂系统动力学研究所、哥廷根大学);Max Planck Institute for Dynamics and Self-Organization, Göttingen(动态与自我组织马克斯·普朗克研究所,哥廷根);Göttingen Campus Institute for Dynamics of Biological Networks, University of Göttingen(哥廷根校区生物网络动力学研究所、哥廷根大学);Institute of Computer Science and Campus Institute Data Science, University of Göttingen(计算机科学研究所和校园数据科学研究所、哥廷根大学)
Compressing Deep Reinforcement Learning Networks with a Dynamic Structured Pruning Method for Autonomous Driving
School of Automation, Guangdong University of Technology(广东工业大学自动化学院);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院)
Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Models
German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Technical University of Darmstadt(达姆施塔特工业大学);Hessian Center for AI (Hessian.AI)(黑森人工智能中心);Centre for Cognitive Science, Technical University of Darmstadt(达姆施塔特工业大学认知科学中心);CISPA Helmholtz Center for Information Security(CISPA亥姆霍兹信息安全中心)