Operational Neural Networks (ONNs) have recently been proposed to address the well-known limitations and drawbacks of conventional Convolutional Neural Networks (CNNs) such as network homogeneity with the sole linear neuron model. ONNs are heterogenous networks with a generalized neuron model that can encapsulate any set of non-linear operators to boost diversity and to learn highly complex and multi-modal functions or spaces with minimal network complexity and training data. However, Greedy Iterative Search (GIS) method, which is the search method used to find optimal operators in ONNs takes many training sessions to find a single operator set per layer. This is not only computationally demanding, but the network heterogeneity is also limited since the same set of operators will then be used for all neurons in each layer. Moreover, the performance of ONNs directly depends on the operator set library used, which introduces a certain risk of performance degradation especially when the optimal operator set required for a particular task is missing from the library. In order to address these issues and achieve an ultimate heterogeneity level to boost the network diversity along with computational efficiency, in this study we propose Self-organized ONNs (Self-ONNs) with generative neurons that have the ability to adapt (optimize) the nodal operator of each connection during the training process. Therefore, Self-ONNs can have an utmost heterogeneity level required by the learning problem at hand. Moreover, this ability voids the need of having a fixed operator set library and the prior operator search within the library in order to find the best possible set of operators. We further formulate the training method to back-propagate the error through the operational layers of Self-ONNs.
SRL-SOA: Self-Representation Learning with Sparse 1D-Operational Autoencoder for Hyperspectral Image Band Selection
Faculty of Information Technology and Communication Sciences, Tampere University(坦佩雷大学信息技术与传播科学学院);Department of Electrical Engineering, Qatar University(卡塔尔大学电气工程学院);Tietoevry Oy(泰托埃弗里公司)
What should a neuron aim for? Designing local objective functions based on information theory
Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen(物理学院、复杂系统动力学研究所、哥廷根大学);Max Planck Institute for Dynamics and Self-Organization, Göttingen(动态与自我组织马克斯·普朗克研究所,哥廷根);Göttingen Campus Institute for Dynamics of Biological Networks, University of Göttingen(哥廷根校区生物网络动力学研究所、哥廷根大学);Institute of Computer Science and Campus Institute Data Science, University of Göttingen(计算机科学研究所和校园数据科学研究所、哥廷根大学)