This paper presents a new artificial neuron model capable of learning its receptive field in the topological domain of inputs. The model provides adaptive and differentiable local connectivity (plasticity) applicable to any domain. It requires no other tool than the backpropagation algorithm to learn its parameters which control the receptive field locations and apertures. This research explores whether this ability makes the neuron focus on informative inputs and yields any advantage over fully connected neurons. The experiments include tests of focusing neuron networks of one or two hidden layers on synthetic and well-known image recognition data sets. The results demonstrated that the focusing neurons can move their receptive fields towards more informative inputs. In the simple two-hidden layer networks, the focusing layers outperformed the dense layers in the classification of the 2D spatial data sets. Moreover, the focusing networks performed better than the dense networks even when 70$\%$ of the weights were pruned. The tests on convolutional networks revealed that using focusing layers instead of dense layers for the classification of convolutional features may work better in some data sets.
Quantifying Local Specialization in Deep Neural Networks
University of California Berkeley(加州大学伯克利分校);Boston University(波士顿大学);Center for Human-Compatible AI (CHAI)(人类兼容人工智能中心);MIT Computer Science and Artificial Intelligence Laboratory (CSAIL)(麻省理工学院计算机科学与人工智能实验室)
Multi-compartment Neuron and Population Encoding Powered Spiking Neural Network for Deep Distributional Reinforcement Learning
Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所类脑认知智能实验室);Center for Long-term Artificial Intelligence(长期人工智能中心);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Chinese Academy of Sciences(中国科学院脑认知与脑智能技术重点实验室)
What should a neuron aim for? Designing local objective functions based on information theory
Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen(物理学院、复杂系统动力学研究所、哥廷根大学);Max Planck Institute for Dynamics and Self-Organization, Göttingen(动态与自我组织马克斯·普朗克研究所,哥廷根);Göttingen Campus Institute for Dynamics of Biological Networks, University of Göttingen(哥廷根校区生物网络动力学研究所、哥廷根大学);Institute of Computer Science and Campus Institute Data Science, University of Göttingen(计算机科学研究所和校园数据科学研究所、哥廷根大学)