Neurons in the brain communicate with spikes, which are discrete events in time and value. Functional network models often employ rate units that are continuously coupled by analog signals. Is there a qualitative difference implied by these two forms of signaling? We develop a unified mean-field theory for large random networks to show that first- and second-order statistics in rate and binary networks are in fact identical if rate neurons receive the right amount of noise. Their response to presented stimuli, however, can be radically different. We quantify these differences by studying how nearby state trajectories evolve over time, asking to what extent the dynamics is chaotic. Chaos in the two models is found to be qualitatively different. In binary networks we find a network-size-dependent transition to chaos and a chaotic submanifold whose dimensionality expands stereotypically with time, while rate networks with matched statistics are nonchaotic. Dimensionality expansion in chaotic binary networks aids classification in reservoir computing and optimal performance is reached within about a single activation per neuron; a fast mechanism for computation that we demonstrate also in spiking networks. A generalization of this mechanism extends to rate networks in their respective chaotic regimes.
Discrete signaling mediates chaotic regularization in recurrent neural networks
The Hebrew University（希伯来大学）；Jülich Research Centre（于利希研究中心）；University College London（伦敦大学学院）；EPFL（洛桑联邦理工学院）；INFN Gruppo Collegato di Parma（帕尔马INFN协作组）；RWTH Aachen University（亚琛工业大学）
It is widely accepted that the complex dynamics characteristic of recurrent neural circuits contributes in a fundamental manner to brain function. Progress has been slow in understanding and exploiting the computational power of recurrent dynamics for two main reasons: nonlinear recurrent networks often exhibit chaotic behavior and most known learning rules do not work in robust fashion in recurrent networks. Here we address both these problems by demonstrating how random recurrent networks (RRN) that initially exhibit chaotic dynamics can be tuned through a supervised learning rule to generat…
A massively recurrent neural network responds on one side to input stimuli and is autonomously active, on the other side, in the absence of sensory inputs. Stimuli and information processing depends crucially on the qualia of the autonomous-state dynamics of the ongoing neural activity. This default neural activity may be dynamically structured in time and space, showing regular, synchronized, bursting or chaotic activity patterns. We study the influence of non-synaptic plasticity on the default dynamical state of recurrent neural networks. The non-synaptic adaption considered acts on intrinsi…
The ability to store and manipulate information is a hallmark of computational systems. Whereas computers are carefully engineered to represent and perform mathematical operations on structured data, neurobiological systems perform analogous functions despite flexible organization and unstructured sensory input. Recent efforts have made progress in modeling the representation and recall of information in neural systems. However, precisely how neural systems learn to modify these representations remains far from understood. Here we demonstrate that a recurrent neural network (RNN) can learn to …
While most models of randomly connected networks assume nodes with simple dynamics, nodes in realistic highly connected networks, such as neurons in the brain, exhibit intrinsic dynamics over multiple timescales. We analyze how the dynamical properties of nodes (such as single neurons) and recurrent connections interact to shape the effective dynamics in large randomly connected networks. A novel dynamical mean-field theory for strongly connected networks of multi-dimensional rate units shows that the power spectrum of the network activity in the chaotic phase emerges from a nonlinear sharpeni…
Princeton University（普林斯顿大学）；Institute for Advanced Study（高等研究院）；CUNY（纽约市立大学）
Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) and neuroscience. Prior theoretical work has focused on RNNs with additive interactions. However, gating - i.e. multiplicative - interactions are ubiquitous in real neurons and also the central feature of the best-performing RNNs in ML. Here, we show that gating offers flexible control of two salient features of the collective dynamics: i) timescales and ii) dimensionality. The gate controlling timescales leads to a novel, marginally stable state, where the network functions as a flexible integ…
Large sparse circuits of spiking neurons exhibit a balanced state of highly irregular activity under a wide range of conditions. It occurs likewise in sparsely connected random networks that receive excitatory external inputs and recurrent inhibition as well as in networks with mixed recurrent inhibition and excitation. Here we analytically investigate this irregular dynamics in finite networks keeping track of all individual spike times and the identities of individual neurons. For delayed, purely inhibitory interactions we show that the irregular dynamics is not chaotic but in fact stable. M…
Advances in machine learning have revolutionized capabilities in applications ranging from natural language processing to marketing to health care. Here, we demonstrate the efficacy of machine learning in predicting chaotic behavior in complex nonlinear mechanical systems. Specifically, we use quasi-recurrent neural networks to predict extremely chaotic time series data obtained from multistable origami systems. Additionally, while machine learning is often viewed as a "black box", in this study we conduct hidden layer analysis to understand how the neural network can process not only periodic…
We study the dynamical states that emerge in a small-world network of recurrently coupled excitable neurons through both numerical and analytical methods. These dynamics depend in large part on the fraction of long-range connections or `short-cuts' and the delay in the neuronal interactions. Persistent activity arises for a small fraction of `short-cuts', while a transition to failure occurs at a critical value of the `short-cut' density. The persistent activity consists of multi-stable periodic attractors, the number of which is at least on the order of the number of neurons in the network. F…
Neuronal activity arises from an interaction between ongoing firing generated spontaneously by neural circuits and responses driven by external stimuli. Using mean-field analysis, we ask how a neural network that intrinsically generates chaotic patterns of activity can remain sensitive to extrinsic input. We find that inputs not only drive network responses, they also actively suppress ongoing activity, ultimately leading to a phase transition in which chaos is completely eliminated. The critical input intensity at the phase transition is a non-monotonic function of stimulus frequency, reveali…
This paper is a review dealing with the study of large size random recurrent neural networks. The connection weights are selected according to a probability law and it is possible to predict the network dynamics at a macroscopic scale using an averaging principle. After a first introductory section, the section 1 reviews the various models from the points of view of the single neuron dynamics and of the global network dynamics. A summary of notations is presented, which is quite helpful for the sequel. In section 2, mean-field dynamics is developed. The probability distribution characterizing …
