We present a first-order non-homogeneous Markov model for the interspike-interval density of a continuously stimulated spiking neuron. The model allows the conditional interspike-interval density and the stationary interspike-interval density to be expressed as products of two separate functions, one of which describes only the neuron characteristics, and the other of which describes only the signal characteristics. This allows the use of this model to predict the response when the underlying neuron model is not known or well determined. The approximation shows particularly clearly that signal autocorrelations and cross-correlations arise as natural features of the interspike-interval density, and are particularly clear for small signals and moderate noise. We show that this model simplifies the design of spiking neuron cross-correlation systems, and describe a four-neuron mutual inhibition network that generates a cross-correlation output for two input signals.
Spiking neural network is a type of artificial neural network in which neurons communicate between each other with spikes. Spikes are identical Boolean events characterized by the time of their arrival. A spiking neuron has internal dynamics and responds to the history of inputs as opposed to the current inputs only. Because of such properties a spiking neural network has rich intrinsic capabilities to process spatiotemporal data. However, because the spikes are discontinuous 'yes or no' events, it is not trivial to apply traditional training procedures such as gradient descend to the spiking …
We demonstrate a spiking neural network for navigation motivated by the chemotaxis network of Caenorhabditis elegans. Our network uses information regarding temporal gradients in the tracking variable's concentration to make navigational decisions. The gradient information is determined by mimicking the underlying mechanisms of the ASE neurons of C. elegans. Simulations show that our model is able to forage and track a target set-point in extremely noisy environments. We develop a VLSI implementation for the main gradient detector neurons, which could be integrated with standard comparator cir…
Network of neurons in the brain apply - unlike processors in our current generation of computer hardware - an event-based processing strategy, where short pulses (spikes) are emitted sparsely by neurons to signal the occurrence of an event at a particular point in time. Such spike-based computations promise to be substantially more power-efficient than traditional clocked processing schemes. However it turned out to be surprisingly difficult to design networks of spiking neurons that are able to carry out demanding computations. We present here a new theoretical framework for organizing comput…
We address the problem of learning feedback control where the controller is a network constructed solely of deterministic spiking neurons. In contrast to previous investigations that were based on a spike rate model of the neuron, the control signal here is determined by the precise temporal positions of spikes generated by the output neurons of the network. We model the problem formally as a hybrid dynamical system comprised of a closed loop between a plant and a spiking neuron network. We derive a novel synaptic weight update rule via which the spiking neuron network controller learns to hol…
This paper describes the Synapto-dendritic Kernel Adapting Neuron (SKAN), a simple spiking neuron model that performs statistical inference and unsupervised learning of spatiotemporal spike patterns. SKAN is the first proposed neuron model to investigate the effects of dynamic synapto-dendritic kernels and demonstrate their computational power even at the single neuron scale. The rule-set defining the neuron is simple there are no complex mathematical operations such as normalization, exponentiation or even multiplication. The functionalities of SKAN emerge from the real-time interaction of si…
Maintaining the ability to fire sparsely is crucial for information encoding in neural networks. Additionally, spiking homeostasis is vital for spiking neural networks with changing numbers of weights and neurons. We discuss a range of network stabilisation approaches, inspired by homeostatic synaptic plasticity mechanisms reported in the brain. These include weight scaling, and weight change as a function of the network's spiking activity. We tested normalisation of the sum of weights for all neurons, and by neuron type. We examined how this approach affects firing rate and performance on clu…
We have added a simplified neuromorphic model of Spike Time Dependent Plasticity (STDP) to the Synapto-dendritic Kernel Adapting Neuron (SKAN). The resulting neuron model is the first to show synaptic encoding of afferent signal to noise ratio in addition to the unsupervised learning of spatio temporal spike patterns. The neuron model is particularly suitable for implementation in digital neuromorphic hardware as it does not use any complex mathematical operations and uses a novel approach to achieve synaptic homeostasis. The neurons noise compensation properties are characterized and tested o…
The effect of environmental temperature on neuronal spiking behaviors is investigated by numerically simulating the temperature dependence of spiking threshold of the Hodgkin-Huxley neuron subject to synaptic stimulus. We find that the spiking threshold exhibits a global minimum in a "comfortable temperature" range where spike initiation needs weakest synaptic strength, indicating the occurrence of optimal use of synaptic transmission in neural system. We further explore the biophysical origin of this phenomenon in ion channel gating kinetics and also discuss its possible biological relevance …
A main concern in cognitive neuroscience is to decode the overt neural spike train observations and infer latent representations under neural circuits. However, traditional methods entail strong prior on network structure and hardly meet the demand for real spike data. Here we propose a novel neural network approach called Neuron Activation Network that extracts neural information explicitly from single trial neuron population spike trains. Our proposed method consists of a spatiotemporal learning procedure on sensory environment and a message passing mechanism on population graph, followed by…
Recurrent neural networks are powerful tools for understanding and modeling computation and representation by populations of neurons. Continuous-variable or "rate" model networks have been analyzed and applied extensively for these purposes. However, neurons fire action potentials, and the discrete nature of spiking is an important feature of neural circuit dynamics. Despite significant advances, training recurrently connected spiking neural networks remains a challenge. We present a procedure for training recurrently connected spiking networks to generate dynamical patterns autonomously, to p…
In machine learning, error back-propagation in multi-layer neural networks (deep learning) has been impressively successful in supervised and reinforcement learning tasks. As a model for learning in the brain, however, deep learning has long been regarded as implausible, since it relies in its basic form on a non-local plasticity rule. To overcome this problem, energy-based models with local contrastive Hebbian learning were proposed and tested on a classification task with networks of rate neurons. We extended this work by implementing and testing such a model with networks of leaky integrate…
Humans and other animals behave as if we perform fast Bayesian inference underlying decisions and movement control given uncertain sense data. Here we show that a biophysically realistic model of the subthreshold membrane potential of a single neuron can exactly compute the numerator in Bayes rule for inferring the Poisson parameter of a sensory spike train. A simple network of spiking neurons can construct and represent the Bayesian posterior density of a parameter of an external cause that affects the Poisson parameter, accurately and in real time.
The response of a neuron to synaptic input strongly depends on whether or not it has just emitted a spike. We propose a neuron model that after spike emission exhibits a partial response to residual input charges and study its collective network dynamics analytically. We uncover a novel desynchronization mechanism that causes a sequential desynchronization transition: In globally coupled neurons an increase in the strength of the partial response induces a sequence of bifurcations from states with large clusters of synchronously firing neurons, through states with smaller clusters to completel…
In this article, we study optimal control problems of spiking neurons whose dynamics are described by a phase model. We design minimum-power current stimuli (controls) that lead to targeted spiking times of neurons, where the cases with unbounded and bounded control amplitude are considered. We show that theoretically the spiking period of a neuron, modeled by phase dynamics, can be arbitrarily altered by a smooth control. However, if the control amplitude is bounded, the range of possible spiking times is constrained and determined by the bound, and feasible spiking times are optimally achiev…
We consider a wide class of spiking neuron models, defined by rather general set of conditions typical for basic models like leaky integrate and fire, or binding neuron model. A neuron is fed with a point renewal process. A relation between the three probability density functions (pdf): (i) pdf of input interspike intervals, (ii) pdf of output interspike intervals of a neuron with instantaneous feedback and (iii) pdf for that same neuron without feedback is derived. This allows to calculate any of the three pdfs provided the another two are given. Similar relation between corresponding means a…
We study the effect of intrinsic heterogeneity on the activity of a population of leaky integrate-and-fire neurons. By rescaling the dynamical equation, we derive mathematical relations between multiple neuronal parameters and a fluctuating input noise. To this end, common input to heterogeneous neurons is conceived as an identical noise with neuron-specific mean and variance. As a consequence, the neuronal output rates can differ considerably, and their relative spike timing becomes desynchronized. This theory can quantitatively explain some recent experimental findings.
There has been a strong push recently to examine biological scale simulations of neuromorphic algorithms to achieve stronger inference capabilities. This paper presents a set of piecewise linear spiking neuron models, which can reproduce different behaviors, similar to the biological neuron, both for a single neuron as well as a network of neurons. The proposed models are investigated, in terms of digital implementation feasibility and costs, targeting large scale hardware implementation. Hardware synthesis and physical implementations on FPGA show that the proposed models can produce precise …
Artificial neural networks built from two-state neurons are powerful computational substrates, whose computational ability is well understood by analogy with statistical mechanics. In this work, we introduce similar analogies in the context of spiking neurons in a fixed time window, where excitatory and inhibitory inputs drawn from a Poisson distribution play the role of temperature. For single neurons with a "bandgap" between their inputs and the spike threshold, this temperature allows for stochastic spiking. By imposing a global inhibitory rhythm over the fixed time windows, we connect neur…
Finite-sized populations of spiking elements are fundamental to brain function, but also used in many areas of physics. Here we present a theory of the dynamics of finite-sized populations of spiking units, based on a quasi-renewal description of neurons with adaptation. We derive an integral equation with colored noise that governs the stochastic dynamics of the population activity in response to time-dependent stimulation and calculate the spectral density in the asynchronous state. We show that systems of coupled populations with adaptation can generate a frequency band in which sensory inf…
Networks of model neurons with balanced recurrent excitation and inhibition produce irregular and asynchronous spiking activity. We extend the analysis of balanced networks to include the known dependence of connection probability on the spatial separation between neurons. In the continuum limit we derive that stable, balanced firing rate solutions require that the spatial spread of external inputs be broader than that of recurrent excitation, which in turn must be broader than or equal to that of recurrent inhibition. For finite size networks we investigate the pattern forming dynamics arisin…
A long tradition in theoretical neuroscience casts sensory processing in the brain as the process of inferring the maximally consistent interpretations of imperfect sensory input. Recently it has been shown that Gamma-band inhibition can enable neural attractor networks to approximately carry out such a sampling mechanism. In this paper we propose a novel neural network model based on irregular gating inhibition, show analytically how it implements a Monte-Carlo Markov Chain (MCMC) sampler, and describe how it can be used to model networks of both neural attractors as well as of single spiking…
Spiking networks that perform probabilistic inference have been proposed both as models of cortical computation and as candidates for solving problems in machine learning. However, the evidence for spike-based computation being in any way superior to non-spiking alternatives remains scarce. We propose that short-term plasticity can provide spiking networks with distinct computational advantages compared to their classical counterparts. In this work, we use networks of leaky integrate-and-fire neurons that are trained to perform both discriminative and generative tasks in their forward and back…
Many central neurons, and in particular certain brainstem aminergic neurons exhibit spontaneous and fairly regular spiking with frequencies of order a few Hz. A large number of ion channel types contribute to such spiking so that accurate modeling of spike generation leads to the requirement of solving very large systems of differential equations, ordinary in the first instance. Since analysis of spiking behavior when many synaptic inputs are active adds further to the number of components, it is useful to have simplified mathematical models of spiking in such neurons so that, for example, sto…
University of Michigan（密歇根大学）；University of Western Australia（西澳大利亚大学）
Spiking neural networks can compensate for quantization error by encoding information either in the temporal domain, or by processing discretized quantities in hidden states of higher precision. In theory, a wide dynamic range state-space enables multiple binarized inputs to be accumulated together, thus improving the representational capacity of individual neurons. This may be achieved by increasing the firing threshold, but make it too high and sparse spike activity turns into no spike emission. In this paper, we propose the use of `threshold annealing' as a warm-up method for firing thresho…
At functional scales, cortical behavior results from the complex interplay of a large number of excitable cells operating in noisy environments. Such systems resist to mathematical analysis, and computational neurosciences have largely relied on heuristic partial (and partially justified) macroscopic models, which successfully reproduced a number of relevant phenomena. The relationship between these macroscopic models and the spiking noisy dynamics of the underlying cells has since then been a great endeavor. Based on recent mean-field reductions for such spiking neurons, we present here {a pr…
Limitations in processing capabilities and memory of today's computers make spiking neuron-based (human) whole-brain simulations inevitably characterized by a compromise between bio-plausibility and computational cost. It translates into brain models composed of a reduced number of neurons and a simplified neuron's mathematical model, leading to the search for new simulation strategies. Taking advantage of the sparse character of brain-like computation, the event-driven technique could represent a way to carry out efficient simulation of large-scale Spiking Neural Networks (SNN). The recent Le…
For a class of excitatory spiking neuron models with delayed feedback fed with a Poisson stochastic process, it is proven that the stream of output interspike intervals cannot be presented as a Markov process of any order. Keywords: spiking neuron; Poisson stochastic process; probability density function; delayed feedback; non-Markov stochastic process
Several studies with brain signals suggested that bottom-up and top-down influences are exerted through distinct frequency bands among visual cortical areas. It has been recently shown that theta and gamma rhythms subserve feedforward, whereas the feedback influence is dominated by the alpha-beta rhythm in primates. A few theoretical models for reproducing these effects have been proposed so far. Here we show that a simple but biophysically plausible two-network motif composed of spiking-neuron models and chemical synapses can exhibit feedforward and feedback influences through distinct freque…
We propose a model of an adaptive network of spiking neurons that gives rise to a hypernetwork of its dynamic states at the upper level of description. Left to itself, the network exhibits a sequence of transient clustering which relates to a traffic in the hypernetwork in the form of a random walk. Receiving inputs the system is able to generate reproducible sequences corresponding to stimulus-specific paths in the hypernetwork. We illustrate these basic notions by a simple network of discrete-time spiking neurons together with its FPGA realization and analyze their properties.
Top-down feedback in cortex is critical for guiding sensory processing, which has prominently been formalized in the theory of hierarchical predictive coding (hPC). However, experimental evidence for error units, which are central to the theory, is inconclusive, and it remains unclear how hPC can be implemented with spiking neurons. To address this, we connect hPC to existing work on efficient coding in balanced networks with lateral inhibition, and predictive computation at apical dendrites. Together, this work points to an efficient implementation of hPC with spiking neurons, where predictio…
The seemingly stochastic transient dynamics of neocortical circuits observed in vivo have been hypothesized to represent a signature of ongoing stochastic inference. In vitro neurons, on the other hand, exhibit a highly deterministic response to various types of stimulation. We show that an ensemble of deterministic leaky integrate-and-fire neurons embedded in a spiking noisy environment can attain the correct firing statistics in order to sample from a well-defined target distribution. We provide an analytical derivation of the activation function on the single cell level; for recurrent netwo…
We establish a general linear response relation for spiking neuronal networks, based on chains with unbounded memory. This relation allows us to predict the influence of a weak amplitude time-dependent external stimuli on spatio-temporal spike correlations, from the spontaneous statistics (without stimulus) in a general context where the memory in spike dynamics can extend arbitrarily far in the past. Using this approach, we show how linear response is explicitly related to neuronal dynamics with an example, the gIF model, introduced by M. Rudolph and A. Destexhe. This example illustrates the …
Spike correlations between neurons are ubiquitous in the cortex, but their role is at present not understood. Here we describe the firing response of a leaky integrate-and-fire neuron (LIF) when it receives a temporarily correlated input generated by presynaptic correlated neuronal populations. Input correlations are characterized in terms of the firing rates, Fano factors, correlation coefficients and correlation timescale of the neurons driving the target neuron. We show that the sum of the presynaptic spike trains cannot be well described by a Poisson process. Solutions of the output firing…
École polytechnique fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Can we use spiking neural networks (SNN) as generative models of multi-neuronal recordings, while taking into account that most neurons are unobserved? Modeling the unobserved neurons with large pools of hidden spiking neurons leads to severely underconstrained problems that are hard to tackle with maximum likelihood estimation. In this work, we use coarse-graining and mean-field approximations to derive a bottom-up, neuronally-grounded latent variable model (neuLVM), where the activity of the unobserved neurons is reduced to a low-dimensional mesoscopic description. In contrast to previous la…
In this paper, we study analytically the impact of an inhibitory autapse on neuronal activity. In order to do this, we formulate conditions on a set of non-adaptive spiking neuron models with delayed feedback inhibition, instead of considering a particular neuronal model. The neuron is stimulated with a stochastic point renewal process of excitatory impulses. Probability density function (PDF) $p(t)$ of output interspike intervals (ISIs) of such a neuron is found exactly without any approximations made. It is expressed in terms of ISIs PDF for the input renewal stream and ISIs PDF for that sam…
Much progress has been made in uncovering the computational capabilities of spiking neural networks. However, spiking neurons will always be more expensive to simulate compared to rate neurons because of the inherent disparity in time scales - the spike duration time is much shorter than the inter-spike time, which is much shorter than any learning time scale. In numerical analysis, this is a classic stiff problem. Spiking neurons are also much more difficult to study analytically. One possible approach to making spiking networks more tractable is to augment mean field activity models with som…
While spike timing has been shown to carry detailed stimulus information at the sensory periphery, its possible role in network computation is less clear. Most models of computation by neural networks are based on population firing rates. In equivalent spiking implementations, firing is assumed to be random such that averaging across populations of neurons recovers the rate-based approach. Recently, however, Denéve and colleagues have suggested that the spiking behavior of neurons may be fundamental to how neuronal networks compute, with precise spike timing determined by each neuron's contrib…
We examine the effects of stochastic input currents on the firing behavior of two excitable neurons coupled with fast excitatory synapses. In such cells (models), typified by the quadratic integrate and fire model, mutual synaptic coupling can cause sustained firing or oscillatory behavior which is necessarily antiphase. Additive Gaussian white noise can transiently terminate the oscillations, hence destroying the stable limit cycle. Further application of the noise may return the system to spiking activity. In a particular noise range, the transition times between the oscillating and the rest…
A major goal of neuroscience, statistical physics and nonlinear dynamics is to understand how brain function arises from the collective dynamics of networks of spiking neurons. This challenge has been chiefly addressed through large-scale numerical simulations. Alternatively, researchers have formulated mean-field theories to gain insight into macroscopic states of large neuronal networks in terms of the collective firing activity of the neurons, or the firing rate. However, these theories have not succeeded in establishing an exact correspondence between the firing rate of the network and the…
In this article, our wish is to demystify some aspects of coding with spike-timing, through a simple review of well-understood technical facts regarding spike coding. The goal is to help better understanding to which extend computing and modelling with spiking neuron networks can be biologically plausible and computationally efficient. We intentionally restrict ourselves to a deterministic dynamics, in this review, and we consider that the dynamics of the network is defined by a non-stochastic mapping. This allows us to stay in a rather simple framework and to propose a review with concrete nu…
We are interested in characterization of synchronization transitions of bursting neurons in the frequency domain. Instantaneous population firing rate (IPFR) $R(t)$, which is directly obtained from the raster plot of neural spikes, is often used as a realistic collective quantity describing population activities in both the computational and the experimental neuroscience. For the case of spiking neurons, a realistic time-domain order parameter, based on $R(t)$, was introduced in our recent work to characterize the spike synchronization transition. Unlike the case of spiking neurons, the IPFR $…
Networks in the brain consist of different types of neurons. Here we investigate the influence of neuron diversity on the dynamics, phase space structure and computational capabilities of spiking neural networks. We find that already a single neuron of a different type can qualitatively change the network dynamics and that mixed networks may combine the computational capabilities of ones with a single neuron type. We study inhibitory networks of concave leaky (LIF) and convex "anti-leaky" (XIF) integrate-and-fire neurons that generalize irregularly spiking non-chaotic LIF neuron networks. Endo…
Starting from a spectral expansion of the Fokker-Plank equation for the membrane potential density in a network of spiking neurons, a low-dimensional dynamics of the collective firing rate is derived. As a result a $n$-order ordinary differential equation for the network activity can be worked out by taking into account the slowest $n$ modes of the expansion. The resulting low-dimensional dynamics naturally takes into account the strength of the synaptic couplings under the hypothesis of a not too fast changing membrane potential density. By considering only the two slowest modes, the firing r…
Spontaneous fluctuations and stimulus response are essential features of neural functioning but how they are connected is poorly understood. I derive fluctuation-dissipation relations (FDR) between the spontaneous spike and voltage correlations and the firing rate susceptibility for i) the leaky integrate-and-fire (IF) model with white noise; ii) an IF model with arbitrary voltage dependence, an adaptation current, and correlated noise. The FDRs can be used to derive correlation statistics or to infer the system's response from observations of its spontaneous activity.
The relationship between a neuron's complex inputs and its spiking output defines the neuron's coding strategy. This is frequently and effectively modeled phenomenologically by one or more linear filters that extract the components of the stimulus that are relevant for triggering spikes, and a nonlinear function that relates stimulus to firing probability. In many sensory systems, these two components of the coding strategy are found to adapt to changes in the statistics of the inputs, in such a way as to improve information transmission. Here, we show for two simple neuron models how feature …
This paper models the dynamics of a large set of interacting neurons within the framework of statistical field theory. We use a method initially developed in the context of statistical field theory [44] and later adapted to complex systems in interaction [45][46]. Our model keeps track of individual interacting neurons dynamics but also preserves some of the features and goals of neural field dynamics, such as indexing a large number of neurons by a space variable. Thus, this paper bridges the scale of individual interacting neurons and the macro-scale modelling of neural field theory.
We investigate the regularity of spontaneous spiking activity on Newman-Watts small-world networks consisting of biophysically realistic Hodgkin-Huxley neurons with a tunable intensity of intrinsic noise and fraction of blocked voltage-gated sodium and potassium ion channels embedded in neuronal membranes. We show that there exists an optimal fraction of shortcut links between physically distant neurons, as well as an optimal intensity of intrinsic noise, which warrant an optimally ordered spontaneous spiking activity. This doubly coherence resonance-like phenomenon depends significantly, and …
The highly variable dynamics of neocortical circuits observed in vivo have been hypothesized to represent a signature of ongoing stochastic inference but stand in apparent contrast to the deterministic response of neurons measured in vitro. Based on a propagation of the membrane autocorrelation across spike bursts, we provide an analytical derivation of the neural activation function that holds for a large parameter space, including the high-conductance state. On this basis, we show how an ensemble of leaky integrate-and-fire neurons with conductance-based synapses embedded in a spiking enviro…
We revisit the dynamics of a prototypical model of balanced activity in networks of spiking neutrons. A detailed investigation of the thermodynamic limit for fixed density of connections (massive coupling) shows that, when inhibition prevails, the asymptotic regime is not asynchronous but rather characterized by a self-sustained irregular, macroscopic (collective) dynamics. So long as the connectivity is massive, this regime is found in many different setups: leaky as well as quadratic integrate-and-fire neurons; large and small coupling strength; weak and strong external currents.
The brain is composed of complex networks of interacting neurons that express considerable heterogeneity in their physiology and spiking characteristics. How does neural heterogeneity affect macroscopic neural dynamics and how does it contribute to neurodynamic functions? In this letter, we address these questions by studying the macroscopic dynamics of networks of heterogeneous Izhikevich neurons. We derive mean-field equations for these networks and examine how heterogeneity in the spiking thresholds of Izhikevich neurons affects the emergent macroscopic dynamics. Our results suggest that th…
