Deep learning is a subset of a broader family of machine learning methods based on learning data representations. These models are inspired by human biological nervous systems, even if there are various differences pertaining to the structural and functional properties of biological brains. The elementary constituents of deep learning models are neurons, which can be considered as functions that receive inputs and produce an output that is a weighted sum of the inputs fed through an activation function. Several models of neurons were proposed in the course of the years that are all based on learnable parameters called weights. In this paper we present a new type of artificial neuron, the double-weight neuron,characterized by additional learnable weights that lead to a more complex and accurate system. We tested a feed-forward and convolutional neural network consisting of double-weight neurons on the MNIST dataset, and we tested a convolution network on the CIFAR-10 dataset. For MNIST we find a $\approx 4\%$ and $\approx 1\%$ improved classification accuracy, respectively, when compared to a standard feed-forward and convolutional neural network built with the same sets of hyperparameters. For CIFAR-10 we find a $\approx 12\%$ improved classification accuracy. We thus conclude that this novel artificial neuron can be considered as a valuable alternative to common ones.
Universidade Federal Rural de Pernambuco（伯南布哥联邦农村大学）；Harvard University（哈佛大学）
The activation function plays a fundamental role in the artificial neural network learning process. However, there is no obvious choice or procedure to determine the best activation function, which depends on the problem. This study proposes a new artificial neuron, named global-local neuron, with a trainable activation function composed of two components, a global and a local. The global component term used here is relative to a mathematical function to describe a general feature present in all problem domain. The local component is a function that can represent a localized behavior, like a t…
No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networks
Department of Data Science, City University of Hong Kong（城市大学数据科学系）；School of Mathematics, Harbin Institute of Technology（哈尔滨工业大学数学系）；School of Instrumentation, Harbin Institute of Technology（哈尔滨工业大学仪器系）；School of Earth and Space Sciences, Peking University（北京大学地球与空间科学学院）；Institute for Advanced Study, Beijing Normal-Hong Kong Baptist University（北京师范大学-香港 Baptist大学高级研究院）
Eindhoven University of Technology（埃因霍温理工大学）；University of Twente（特文特大学）；Iowa State University（爱荷华州立大学）
Artificial Neural Networks (ANNs) have emerged as hot topics in the research community. Despite the success of ANNs, it is challenging to train and deploy modern ANNs on commodity hardware due to the ever-increasing model size and the unprecedented growth in the data volumes. Particularly for microarray data, the very-high dimensionality and the small number of samples make it difficult for machine learning techniques to handle. Furthermore, specialized hardware such as Graphics Processing Unit (GPU) is expensive. Sparse neural networks are the leading approaches to address these challenges. H…
Bio-Inspired Adaptive Neurons for Dynamic Weighting in Artificial Neural Networks
organization= College Of Science \& Engineering, Hamad Bin Khalifa University , addressline= Education City , city= Doha , postcode= 34110 , country= Qatar；organization= School of Electrical, Computer \& Telecommunications Engineering, University of Wollongong , addressline= Address Two , city= Wollongong , postcode= 2500 , state= New South Wales , country= Australia
Neuronal diversity can improve machine learning for physics and beyond
North Carolina State University（北卡罗来纳州立大学）；The Jackson Laboratory for Genomic Medicine（杰克逊基因组医学实验室）；The College of Wooster（伍斯特学院）；Indian Institute of Science Education and Research Mohali（印度科学教育与研究学院莫哈利分校）
Advanced Research Center, National Fusion Research Institute (NFRI)（国家聚变研究所高级研究中心）
Inspired by recent theoretical arguments, physics-informed echo state network (ESN) is discussed on the attempt to train a reservoir model absolutely in physics-informed manner. As the plainest work on such a purpose, an ODE (ordinary differential equation) approximator is designed to replicate the solution in sequence with respect to the recurrent evaluations. On the principal invariance of differential equations, the constraint in recurrence just takes shape to secure a proper regression method for the ESN-based ODE approximator. After then, the actual training process is established on the …
Deep neural networks (DNNs) are known for extracting useful information from large amounts of data. However, the representations learned in DNNs are typically hard to interpret, especially in dense layers. One crucial issue of the classical DNN model such as multilayer perceptron (MLP) is that neurons in the same layer of DNNs are conditionally independent of each other, which makes co-training and emergence of higher modularity difficult. In contrast to DNNs, biological neurons in mammalian brains display substantial dependency patterns. Specifically, biological neural networks encode represe…
Deep neural networks have achieved human-level accuracy on almost all perceptual benchmarks. It is interesting that these advances were made using two ideas that are decades old: (a) an artificial neuron based on a linear summator and (b) SGD training. However, there are important metrics beyond accuracy: computational efficiency and stability against adversarial perturbations. In this paper, we propose two closely connected methods to improve these metrics on contour recognition tasks: (a) a novel model of an artificial neuron, a "strong neuron," with low hardware requirements and inherent ro…
The Chinese University of Hong Kong（香港中文大学）；Southeast University（东南大学）；Cornell University（康奈尔大学）
Throughout history, the development of artificial intelligence, particularly artificial neural networks, has been open to and constantly inspired by the increasingly deepened understanding of the brain, such as the inspiration of neocognitron, which is the pioneering work of convolutional neural networks. Per the motives of the emerging field: NeuroAI, a great amount of neuroscience knowledge can help catalyze the next generation of AI by endowing a network with more powerful capabilities. As we know, the human brain has numerous morphologically and functionally different neurons, while artifi…
In this work, we formulate NEWRON: a generalization of the McCulloch-Pitts neuron structure. This new framework aims to explore additional desirable properties of artificial neurons. We show that some specializations of NEWRON allow the network to be interpretable with no change in their expressiveness. By just inspecting the models produced by our NEWRON-based networks, we can understand the rules governing the task. Extensive experiments show that the quality of the generated models is better than traditional interpretable models and in line or better than standard neural networks.
这些讲义是为那些刚开始接触人工神经网络的人设计的。我们展示了它是如何工作的一个单一的人工神经元（麦卡洛克和皮特模型），数学和图形。我们解释了delta规则，这是一种找到神经元权重的学习算法。我们还介绍了MATLAB/Octave中的一些例子。有线性和非线性问题的分类任务的例子。最后，我们提出了一个人工神经网络，一个前馈神经网络及其学习算法反向传播。- Escherapuntes están diseñados para personas que por primera vez se introducen en en el tema de las redes neuronales artificiales.它必须具有神经元的基本功能，数学和语法。在Regla Delta中，算法用于计算一个神经元的比索。También se muestran ejemplos en MATLAB/Octave. Hay ejemplos para problemas de clasificación，para problemas lineales y no-lineales.最后，我们采用反向传播的方法构建了红色神经元人工芋螺。
Aston Institute of Photonic Technologies（阿斯顿光子技术研究所）；Aston University（阿斯顿大学）
Many modern applications of the artificial neural networks ensue large number of layers making traditional digital implementations increasingly complex. Optical neural networks offer parallel processing at high bandwidth, but have the challenge of noise accumulation. We propose here a new type of neural networks using stochastic resonances as an inherent part of the architecture and demonstrate a possibility of significant reduction of the required number of neurons for a given performance accuracy. We also show that such a neural network is more robust against the impact of noise.
Neuromorphic computing with spintronic devices has been of interest due to the limitations of CMOS-driven von Neumann computing. Domain wall-magnetic tunnel junction (DW-MTJ) devices have been shown to be able to intrinsically capture biological neuron behavior. Edgy-relaxed behavior, where a frequently firing neuron experiences a lower action potential threshold, may provide additional artificial neuronal functionality when executing repeated tasks. In this study, we demonstrate that this behavior can be implemented in DW-MTJ artificial neurons via three alternative mechanisms: shape anisotro…
How Do Artificial Intelligences Think? The Three Mathematico-Cognitive Factors of Categorical Segmentation Operated by Synthetic Neurons
Neocognition & Faculties of Philosophy and Psychology, Paris（巴黎新认知与哲学心理学学院）；Neocognition and NP - Phelma, UGA（佐治亚大学新认知与NP-Phelma学院）；Neocognition（新认知机构）；Neocognition and Faculty of Medicine of Lyon East, University Lyon 1（里昂第一大学里昂东部医学院新认知机构）；Neocognition (Chrysippe R&D) and Department of Psychology, University of Paris 8（巴黎第八大学心理学系新认知机构（克吕西普研发中心））；Neocognition (Chrysippe R&D) and Epitech Paris（巴黎Epitech新认知机构（克吕西普研发中心））
In the quest for low power, bio-inspired computation both memristive and memcapacitive-based Artificial Neural Networks (ANN) have been the subjects of increasing focus for hardware implementation of neuromorphic computing. One step further, regenerative capacitive neural networks, which call for the use of adiabatic computing, offer a tantalising route towards even lower energy consumption, especially when combined with `memimpedace' elements. Here, we present an artificial neuron featuring adiabatic synapse capacitors to produce membrane potentials for the somas of neurons; the latter implem…
Updating the standard neuron model in artificial neural networks
Spanish National Research Council (CSIC)（西班牙国家研究理事会（CSIC））；Center for Research in Mathematics (CIMAT)（数学研究中心（CIMAT））；Universidad Autónoma de Madrid (UAM)（马德里自治大学（UAM））；National Science Foundation (NSF)（国家科学基金会（NSF））
Excitable optoelectronic devices represent one of the key building blocks for implementation of artificial spiking neurons in neuromorphic (brain-inspired) photonic systems. This work introduces and experimentally investigates an opto-electro-optical (O/E/O) artificial neuron built with a resonant tunnelling diode (RTD) coupled to a photodetector as a receiver and a vertical cavity surface emitting laser as a the transmitter. We demonstrate a well defined excitability threshold, above which this neuron produces 100 ns optical spiking responses with characteristic neural-like refractory period.…
Artificial spiking neural networks have found applications in areas where the temporal nature of activation offers an advantage, such as time series prediction and signal processing. To improve their efficiency, spiking architectures often run on custom-designed neuromorphic hardware, but, despite their attractive properties, these implementations have been limited to digital systems. We describe an artificial quantum spiking neuron that relies on the dynamical evolution of two easy to implement Hamiltonians and subsequent local measurements. The architecture allows exploiting complex amplitud…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）
The spiking neural network (SNN) computes and communicates information through discrete binary events. It is considered more biologically plausible and more energy-efficient than artificial neural networks (ANN) in emerging neuromorphic hardware. However, due to the discontinuous and non-differentiable characteristics, training SNN is a relatively challenging task. Recent work has achieved essential progress on an excellent performance by converting ANN to SNN. Due to the difference in information processing, the converted deep SNN usually suffers serious performance loss and large time delay.…
Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Florida（佛罗里达大学）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）
Inspired by neuronal diversity in the biological neural system, a plethora of studies proposed to design novel types of artificial neurons and introduce neuronal diversity into artificial neural networks. Recently proposed quadratic neuron, which replaces the inner-product operation in conventional neurons with a quadratic one, have achieved great success in many essential tasks. Despite the promising results of quadratic neurons, there is still an unresolved issue: \textit{Is the superior performance of quadratic networks simply due to the increased parameters or due to the intrinsic expressi…
Identification of different neuronal cell types is critical for understanding their contribution to brain functions. Yet, automated and reliable classification of neurons remains a challenge, primarily because of their biological complexity. Typical approaches include laborious and expensive immunohistochemical analysis while feature extraction algorithms based on cellular characteristics have recently been proposed. The former rely on molecular markers, which are often expressed in many cell types, while the latter suffer from similar issues: finding features that are distinctive for each cla…
Fast gradient-free activation maximization for neurons in spiking neural networks
Laboratory of Neuronal Intelligence, Institute for Advanced Brain Studies, Lomonosov Moscow State University（神经智能实验室，先进脑科学研究所，洛蒙诺索夫莫斯科国立大学）；Artificial Intelligence Research Institute (AIRI)（人工智能研究所）；Skolkovo Institute of Science and Technology（斯克罗夫科技研究院）；HSE University Moscow, Russia（莫斯科高等经济大学，俄罗斯）
Complementary metal oxide semiconductor (CMOS) devices display volatile characteristics, and are not well suited for analog applications such as neuromorphic computing. Spintronic devices, on the other hand, exhibit both non-volatile and analog features, which are well-suited to neuromorphic computing. Consequently, these novel devices are at the forefront of beyond-CMOS artificial intelligence applications. However, a large quantity of these artificial neuromorphic devices still require the use of CMOS, which decreases the efficiency of the system. To resolve this, we have previously proposed…
We consider Hodgkin-Huxley-type model that is a stiff ODE system with two fast and one slow variables. For the parameter ranges under consideration the original version of the model has unstable fixed point and the oscillating attractor that demonstrates bifurcation from bursting to spiking dynamics. Also a modified version is considered where the bistability occurs such that an area in the parameter space appears where the fixed point becomes stable and coexists with the bursting attractor. For these two systems we create artificial neural networks that are able to reproduce their dynamics. T…
In this paper, we are introducing a novel model of artificial intelligence, the functional neural network for modeling of human decision-making processes. This neural network is composed of multiple artificial neurons racing in the network. Each of these neurons has a similar structure programmed independently by the users and composed of an intention wheel, a motor core and a sensory core representing the user itself and racing at a specific velocity. The mathematics of the neuron's formulation and the racing mechanism of multiple nodes in the network will be discussed, and the group decision…
The neuromorphic BrainScaleS-2 ASIC comprises mixed-signal neurons and synapse circuits as well as two versatile digital microprocessors. Primarily designed to emulate spiking neural networks, the system can also operate in a vector-matrix multiplication and accumulation mode for artificial neural networks. Analog multiplication is carried out in the synapse circuits, while the results are accumulated on the neurons' membrane capacitors. Designed as an analog, in-memory computing device, it promises high energy efficiency. Fixed-pattern noise and trial-to-trial variations, however, require the…
