Although conditional branching between possible behavioural states is a hallmark of intelligent behavior, very little is known about the neuronal mechanisms that support this processing. In a step toward solving this problem we demonstrate by theoretical analysis and simulation how networks of richly inter-connected neurons, such as those observed in the superficial layers of the neocortex, can embed reliable robust finite state machines. We show how a multi-stable neuronal network containing a number of states can be created very simply, by coupling two recurrent networks whose synaptic weights have been configured for soft winner-take-all (sWTA) performance. These two sWTAs have simple, homogenous locally recurrent connectivity except for a small fraction of recurrent cross-connections between them, which are used to embed the required states. This coupling between the maps allows the network to continue to express the current state even after the input that elicted that state is withdrawn. In addition, a small number of 'transition neurons' implement the necessary input-driven transitions between the embedded states. We provide simple rules to systematically design and construct neuronal state machines of this kind. The significance of our finding is that it offers a method whereby the cortex could construct networks supporting a broad range of sophisticated processing by applying only small specializations to the same generic neuronal circuit.
We present and analyze deterministic complex networks of pulse-coupled oscillators that exhibits recurrent events comprised of an increase and a decline in synchrony. Events emerging from the networks may form an oscillatory behavior or may be separated by periods of asynchrony with varying duration. The phenomenon is specific to spatial networks with both short- and long-ranged connections and requires delayed interactions and refractoriness of oscillators.
Randomly connected networks of excitatory and inhibitory spiking neurons provide a parsimonious model of neural variability, but are notoriously unreliable for performing computations. We show that this difficulty is overcome by incorporating the well-documented dependence of connection probability on distance. Spatially extended spiking networks exhibit symmetry-breaking bifurcations and generate spatiotemporal patterns that can be trained to perform dynamical computations.
Recurrent neural network architectures can have useful computational properties, with complex temporal dynamics and input-sensitive attractor states. However, evaluation of recurrent dynamic architectures requires solution of systems of differential equations, and the number of evaluations required to determine their response to a given input can vary with the input, or can be indeterminate altogether in the case of oscillations or instability. In feed-forward networks, by contrast, only a single pass through the network is needed to determine the response to a given input. Modern machine-lear…
Mean-field systems have been previously derived for networks of coupled, two-dimensional, integrate-and-fire neurons such as the Izhikevich, adapting exponential (AdEx) and quartic integrate and fire (QIF), among others. Unfortunately, the mean-field systems have a degree of frequency error and the networks analyzed often do not include noise when there is adaptation. Here, we derive a one-dimensional partial differential equation (PDE) approximation for the marginal voltage density under a first order moment closure for coupled networks of integrate-and-fire neurons with white noise inputs. T…
Large scale neural recordings have established that the transformation of sensory stimuli into motor outputs relies on low-dimensional dynamics at the population level, while individual neurons exhibit complex selectivity. Understanding how low-dimensional computations on mixed, distributed representations emerge from the structure of the recurrent connectivity and inputs to cortical networks is a major challenge. Here, we study a class of recurrent network models in which the connectivity is a sum of a random part and a minimal, low-dimensional structure. We show that, in such networks, the d…
Recognition of objects from partial information presents a significant challenge for theories of vision because it requires spatial integration and extrapolation from prior knowledge. We combined neurophysiological recordings in human cortex with psychophysical measurements and computational modeling to investigate the mechanisms involved in object completion. We recorded intracranial field potentials from 1,699 electrodes in 18 epilepsy patients to measure the timing and selectivity of responses along human visual cortex to whole and partial objects. Responses along the ventral visual stream …
Neural networks are complex algorithms that loosely model the behaviour of the human brain. They play a significant role in computational neuroscience and artificial intelligence. The next generation of neural network models is based on the spike timing activity of neurons: spiking neural networks (SNNs). However, model parameters in SNNs are difficult to search and optimise. Previous studies using genetic algorithm (GA) optimisation of SNNs were focused mainly on simple, feedforward, or oscillatory networks, but not much work has been done on optimising cortex-like recurrent SNNs. In this wor…
Harvard University（哈佛大学）；Augsburg University（奥格斯堡大学）；Technische Universität München（慕尼黑工业大学）；Ludwig-Maximilians-Universität München（慕尼黑大学）
Making inferences from partial information constitutes a critical aspect of cognition. During visual perception, pattern completion enables recognition of poorly visible or occluded objects. We combined psychophysics, physiology and computational models to test the hypothesis that pattern completion is implemented by recurrent computations and present three pieces of evidence that are consistent with this hypothesis. First, subjects robustly recognized objects even when rendered <15% visible, but recognition was largely impaired when processing was interrupted by backward masking. Second, inva…
Cortical circuits exhibit intricate recurrent architectures that are remarkably similar across different brain areas. Such stereotyped structure suggests the existence of common computational principles. However, such principles have remained largely elusive. Inspired by gated-memory networks, namely long short-term memory networks (LSTMs), we introduce a recurrent neural network in which information is gated through inhibitory cells that are subtractive (subLSTM). We propose a natural mapping of subLSTMs onto known canonical excitatory-inhibitory cortical microcircuits. Our empirical evaluati…
Center for Quantitative Biology, Academy for Advanced Interdisciplinary Studies, Peking University（北京大学前沿交叉学科研究院定量生物学中心）；School of Physics, Peking University（北京大学物理学院）；Center for Bioinformatics, National Laboratory of Protein Engineering and Plant Genetic Engineering, School of Life Sciences and Center for Quantitative Biology, Academy for Advanced Interdisciplinary Studies,Peking University（北京大学生命科学学院、前沿交叉学科研究院定量生物学中心及蛋白质工程与植物基因工程国家重点实验室生物信息中心）；CNRS, LJAD and Institute NeuroMod, Université Côte d’Azur（法国蔚蓝海岸大学神经调节研究所、LJAD及法国国家科学研究中心）
Anatomical studies demonstrate that brain reformats input information to generate reliable responses for performing computations. However, it remains unclear how neural circuits encode complex spatio-temporal patterns. We show that neural dynamics are strongly influenced by the phase alignment between the input and the spontaneous chaotic activity. Input structuring along the dominant chaotic projections causes the chaotic trajectories to become stable channels (or attractors), hence, improving the computational capability of a recurrent network. Using mean field analysis, we derive the impact…
Laboratory of Biological Modeling, NIDDK/NIH（美国国立糖尿病、消化系统疾病和肾病研究所（NIDDK）/美国国立卫生研究院（NIH）生物建模实验室）
Spiking activity of neurons engaged in learning and performing a task show complex spatiotemporal dynamics. While the output of recurrent network models can learn to perform various tasks, the possible range of recurrent dynamics that emerge after learning remains unknown. Here we show that modifying the recurrent connectivity with a recursive least squares algorithm provides sufficient flexibility for synaptic and spiking rate dynamics of spiking networks to produce a wide range of spatiotemporal activity. We apply the training method to learn arbitrary firing patterns, stabilize irregular sp…
Recurrent neural networks (RNNs) are widely used in computational neuroscience and machine learning applications. In an RNN, each neuron computes its output as a nonlinear function of its integrated input. While the importance of RNNs, especially as models of brain processing, is undisputed, it is also widely acknowledged that the computations in standard RNN models may be an over-simplification of what real neuronal networks compute. Here, we suggest that the RNN approach may be made both neurobiologically more plausible and computationally more powerful by its fusion with Bayesian inference …
We analyze zero-lag and cluster synchrony of delay-coupled non-smooth dynamical systems by extending the master stability approach, and apply this to networks of adaptive threshold-model neurons. For a homogeneous population of excitatory and inhibitory neurons we find (i) that subthreshold adaptation stabilizes or destabilizes synchrony depending on whether the recurrent synaptic excitatory or inhibitory couplings dominate, and (ii) that synchrony is always unstable for networks with balanced recurrent synaptic inputs. If couplings are not too strong, synchronization properties are similar fo…
Zuckerman Mind Brain Behavior Institute, Columbia University（哥伦比亚大学扎克曼心脑行为研究所）；Columbia University（哥伦比亚大学）
Biological visual systems exhibit abundant recurrent connectivity. State-of-the-art neural network models for visual recognition, by contrast, rely heavily or exclusively on feedforward computation. Any finite-time recurrent neural network (RNN) can be unrolled along time to yield an equivalent feedforward neural network (FNN). This important insight suggests that computational neuroscientists may not need to engage recurrent computation, and that computer-vision engineers may be limiting themselves to a special case of FNN if they build recurrent models. Here we argue, to the contrary, that F…
Recurrent neural networks are powerful tools for understanding and modeling computation and representation by populations of neurons. Continuous-variable or "rate" model networks have been analyzed and applied extensively for these purposes. However, neurons fire action potentials, and the discrete nature of spiking is an important feature of neural circuit dynamics. Despite significant advances, training recurrently connected spiking neural networks remains a challenge. We present a procedure for training recurrently connected spiking networks to generate dynamical patterns autonomously, to p…
Networks of strongly-coupled neurons with random connectivity exhibit chaotic, asynchronous fluctuations. In previous work, we showed that when endowed with an additional low-rank connectivity consisting of the outer product of orthogonal vectors, these networks generate large-scale coherent fluctuations. Although a striking phenomenon, that result depended on a fine-tuned choice of low-rank structure. Here we extend that work by generalizing the theory of excitation-inhibition balance to networks with arbitrary low-rank structure and show that low-dimensional variability emerges intrinsically…
The computational properties of neural systems are often thought to be implemented in terms of their network dynamics. Hence, recovering the system dynamics from experimentally observed neuronal time series, like multiple single-unit (MSU) recordings or neuroimaging data, is an important step toward understanding its computations. Ideally, one would not only seek a state space representation of the dynamics, but would wish to have access to its governing equations for in-depth analysis. Recurrent neural networks (RNNs) are a computationally powerful and dynamically universal formal framework w…
Oscillatory dynamics are ubiquitous in biological networks. Possible sources of oscillations are well understood in low-dimensional systems, but have not been fully explored in high-dimensional networks. Here we study large networks consisting of randomly coupled rate units. We identify a novel type of bifurcation in which a continuous part of the eigenvalue spectrum of the linear stability matrix crosses the instability line at non-zero-frequency. This bifurcation occurs when the interactions are delayed and partially anti-symmetric, and leads to a heterogeneous oscillatory state in which osc…
Recurrent neural networks are extremely powerful yet hard to train. One of their issues is the vanishing gradient problem, whereby propagation of training signals may be exponentially attenuated, freezing training. Use of orthogonal or unitary matrices, whose powers neither explode nor decay, has been proposed to mitigate this issue, but their computational expense has hindered their use. Here we show that in the specific case of convolutional RNNs, we can define a convolutional exponential and that this operation transforms antisymmetric or anti-Hermitian convolution kernels into orthogonal o…
It is widely accepted that the complex dynamics characteristic of recurrent neural circuits contributes in a fundamental manner to brain function. Progress has been slow in understanding and exploiting the computational power of recurrent dynamics for two main reasons: nonlinear recurrent networks often exhibit chaotic behavior and most known learning rules do not work in robust fashion in recurrent networks. Here we address both these problems by demonstrating how random recurrent networks (RRN) that initially exhibit chaotic dynamics can be tuned through a supervised learning rule to generat…
We study the reliability of spike output in a general class of pulse-coupled oscillators receiving a fluctuating input. Showing that this problem is equivalent to noise-induced synchronization between identical networks of oscillators, we employ the phase reduction method to analytically derive the average Lyapunov exponent of the synchronized state. We show that a transition occurs between reliable and unreliable responses at a critical coupling strength, which is determined through the competition between the external input and recurrent input. To our surprise, the critical value does not de…
