Although there is increasing evidence of criticality in the brain, the processes that guide neuronal networks to reach or maintain criticality remain unclear. The present research examines the role of neuronal gain plasticity in time-series of simulated neuronal networks composed of integrate-and-fire stochastic spiking neurons, and the utility of fractal methods in assessing network criticality. Simulated time-series were derived from a network model of fully connected discrete-time stochastic excitable neurons. Monofractal and multifractal analyses were applied to neuronal gain time-series. Fractal scaling was greatest in networks with a mid-range of neuronal plasticity, versus extremely high or low levels of plasticity. Peak fractal scaling corresponded closely to additional indices of criticality, including average branching ratio. Networks exhibited multifractal structure, or multiple scaling relationships. Multifractal spectra around peak criticality exhibited elongated right tails, suggesting that the fractal structure is relatively insensitive to high-amplitude local fluctuations. Networks near critical states exhibited mid-range multifractal spectra width and tail length, which is consistent with literature suggesting that networks poised at quasi-critical states must be stable enough to maintain organization but unstable enough to be adaptable. Lastly, fractal analyses may offer additional information about critical state dynamics of networks by indicating scales of influence as networks approach critical states.
The collective dynamics of neural populations are often characterized in terms of correlations in the spike activity of different neurons. Open questions surround the basic nature of these correlations. In particular, what leads to higher-order correlations -- correlations in the population activity that extend beyond those expected from cell pairs? Here, we examine this question for a simple, but ubiquitous, circuit feature: common fluctuating input arriving to spiking neurons of integrate-and-fire type. We show that leads to strong higher-order correlations, as for earlier work with discrete…
We consider a threshold-crossing spiking process as a simple model for the activity within a population of neurons. Assuming that these neurons are driven by a common fluctuating input with Gaussian statistics, we evaluate the cross-correlation of spike trains in pairs of model neurons with different thresholds. This correlation function tends to be asymmetric in time, indicating a preference for the neuron with the lower threshold to fire before the one with the higher threshold, even if their inputs are identical. The relationship between these results and spike statistics in other models of…
Harmonic fractal transformation for modeling complex neuronal effects: from bursting and noise shaping to waveform sensitivity and noise-induced subthreshold spiking
Collective oscillations and their suppression by external stimulation are analyzed in a large-scale neural network consisting of two interacting populations of excitatory and inhibitory quadratic integrate-and-fire neurons. In the limit of an infinite number of neurons, the microscopic model of this network can be reduced to an exact low-dimensional system of mean-field equations. Bifurcation analysis of these equations reveals three different dynamic modes in a free network: a stable resting state, a stable limit cycle, and bistability with a coexisting resting state and a limit cycle. We sho…
Voltage-sensitive dye imaging (VSDi) has revealed fundamental properties of neocortical processing at mesoscopic scales. Since VSDi signals report the average membrane potential, it seems natural to use a mean-field formalism to model such signals. Here, we investigate a mean-field model of networks of Adaptive Exponential (AdEx) integrate-and-fire neurons, with conductance-based synaptic interactions. The AdEx model can capture the spiking response of different cell types, such as regular-spiking (RS) excitatory neurons and fast-spiking (FS) inhibitory neurons. We use a Master Equation formal…
This paper proposes a methodology to extract a low-dimensional integrate-and-fire model from an arbitrarily detailed single-compartment biophysical model. The method aims at relating the modulation of maximal conductance parameters in the biophysical model to the modulation of parameters in the proposed integrate-and-fire model. The approach is illustrated on two well-documented examples of cellular neuromodulation: the transition between Type I and Type II excitability and the transition between spiking and bursting.
A common approach to interpreting spiking activity is based on identifying the firing fields---regions in physical or configuration spaces that elicit responses of neurons. Common examples include hippocampal place cells that fire at preferred locations in the navigated environment, head direction cells that fire at preferred orientations of the animal's head, view cells that respond to preferred spots in the visual field, etc. In all these cases, firing fields were discovered empirically, by trial and error. We argue that the existence and a number of properties of the firing fields can be es…
We study the effect of intrinsic heterogeneity on the activity of a population of leaky integrate-and-fire neurons. By rescaling the dynamical equation, we derive mathematical relations between multiple neuronal parameters and a fluctuating input noise. To this end, common input to heterogeneous neurons is conceived as an identical noise with neuron-specific mean and variance. As a consequence, the neuronal output rates can differ considerably, and their relative spike timing becomes desynchronized. This theory can quantitatively explain some recent experimental findings.
The Bayesian view of the brain hypothesizes that the brain constructs a generative model of the world, and uses it to make inferences via Bayes' rule. Although many types of approximate inference schemes have been proposed for hierarchical Bayesian models of the brain, the questions of how these distinct inference procedures can be realized by hierarchical networks of spiking neurons remains largely unresolved. Based on a previously proposed multi-compartment neuron model in which dendrites perform logarithmic compression, and stochastic spiking winner-take-all (WTA) circuits in which firing p…
Finite-sized populations of spiking elements are fundamental to brain function, but also used in many areas of physics. Here we present a theory of the dynamics of finite-sized populations of spiking units, based on a quasi-renewal description of neurons with adaptation. We derive an integral equation with colored noise that governs the stochastic dynamics of the population activity in response to time-dependent stimulation and calculate the spectral density in the asynchronous state. We show that systems of coupled populations with adaptation can generate a frequency band in which sensory inf…
Artificial neural networks (ANNs) have been extensively used for the description of problems arising from biological systems and for constructing neuromorphic computing models. The third generation of ANNs, namely, spiking neural networks (SNNs), inspired by biological neurons enable a more realistic mimicry of the human brain. A large class of the problems from these domains is characterized by the necessity to deal with the combination of neurons, spikes and synapses via integrate-and-fire neuron models. Motivated by important applications of the integrate-and-fire of neurons in neuromorphic…
One of the fundamental characteristics of a nonlinear system is how it transfers correlations in its inputs to correlations in its outputs. This is particularly important in the nervous system, where correlations between spiking neurons are prominent. Using linear response and asymptotic methods for pairs of unconnected integrate-and-fire (IF) neurons receiving white noise inputs, we show that this correlation transfer depends on the output spike firing rate in a strong, stereotyped manner, and is, surprisingly, almost independent of the interspike variance. For cells receiving heterogeneous i…
The background activity of a cortical neural network is modeled by a homogeneous integrate-and-fire network with unreliable inhibitory synapses. Numerical and analytical calculations show that the network relaxes into a stationary state of high attention. The majority of the neurons has a membrane potential just below the threshold; as a consequence the network can react immediately - on the time scale of synaptic transmission- on external pulses. The neurons fire with a low rate and with a broad distribution of interspike intervals. Firing events of the total network are correlated over short…
A neuron transforms its input into output spikes, and this transformation is the basic unit of computation in the nervous system. The spiking response of the neuron to a complex, time-varying input can be predicted from the detailed biophysical properties of the neuron, modeled as a deterministic nonlinear dynamical system. In the tradition of neural coding, however, a neuron or neural system is treated as a black box and statistical techniques are used to identify functional models of its encoding properties. The goal of this work is to connect the mechanistic, biophysical approach to neurona…
We present a simple Markov model of spiking neural dynamics that can be analytically solved to characterize the stochastic dynamics of a finite-size spiking neural network. We give closed-form estimates for the equilibrium distribution, mean rate, variance and autocorrelation function of the network activity. The model is applicable to any network where the probability of firing of a neuron in the network only depends on the number of neurons that fired in a previous temporal epoch. Networks with statistically homogeneous connectivity and membrane and synaptic time constants that are not exces…
While spike timing has been shown to carry detailed stimulus information at the sensory periphery, its possible role in network computation is less clear. Most models of computation by neural networks are based on population firing rates. In equivalent spiking implementations, firing is assumed to be random such that averaging across populations of neurons recovers the rate-based approach. Recently, however, Denéve and colleagues have suggested that the spiking behavior of neurons may be fundamental to how neuronal networks compute, with precise spike timing determined by each neuron's contrib…
We consider a wide class of spiking neuron models, defined by rather general set of conditions typical for basic models like leaky integrate and fire, or binding neuron model. A neuron is fed with a point renewal process. A relation between the three probability density functions (pdf): (i) pdf of input interspike intervals, (ii) pdf of output interspike intervals of a neuron with instantaneous feedback and (iii) pdf for that same neuron without feedback is derived. This allows to calculate any of the three pdfs provided the another two are given. Similar relation between corresponding means a…
Recurrent neural networks are powerful tools for understanding and modeling computation and representation by populations of neurons. Continuous-variable or "rate" model networks have been analyzed and applied extensively for these purposes. However, neurons fire action potentials, and the discrete nature of spiking is an important feature of neural circuit dynamics. Despite significant advances, training recurrently connected spiking neural networks remains a challenge. We present a procedure for training recurrently connected spiking networks to generate dynamical patterns autonomously, to p…
We examine the effects of stochastic input currents on the firing behavior of two excitable neurons coupled with fast excitatory synapses. In such cells (models), typified by the quadratic integrate and fire model, mutual synaptic coupling can cause sustained firing or oscillatory behavior which is necessarily antiphase. Additive Gaussian white noise can transiently terminate the oscillations, hence destroying the stable limit cycle. Further application of the noise may return the system to spiking activity. In a particular noise range, the transition times between the oscillating and the rest…
ILIF: Temporal Inhibitory Leaky Integrate-and-Fire Neuron for Overactivation in Spiking Neural Networks
Department of Data Science and AI, Monash University（墨尔本大学数据科学与人工智能系）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；School of Software, Northeastern University（东北大学软件学院）
We propose a new supervised learning rule for multilayer spiking neural networks (SNNs) that use a form of temporal coding known as rank-order-coding. With this coding scheme, all neurons fire exactly one spike per stimulus, but the firing order carries information. In particular, in the readout layer, the first neuron to fire determines the class of the stimulus. We derive a new learning rule for this sort of network, named S4NN, akin to traditional error backpropagation, yet based on latencies. We show how approximated error gradients can be computed backward in a feedforward network with an…
The spiking activity of single neurons can be well described by a nonlinear integrate-and-fire model that includes somatic adaptation. When exposed to fluctuating inputs sparsely coupled populations of these model neurons exhibit stochastic collective dynamics that can be effectively characterized using the Fokker-Planck equation. [...] Here we derive from that description four simple models for the spike rate dynamics in terms of low-dimensional ordinary differential equations using two different reduction techniques: one uses the spectral decomposition of the Fokker-Planck operator, the othe…
The design and analysis of spiking neural network algorithms will be accelerated by the advent of new theoretical approaches. In an attempt at such approach, we provide a principled derivation of a spiking algorithm for unsupervised learning, starting from the nonnegative similarity matching cost function. The resulting network consists of integrate-and-fire units and exhibits local learning rules, making it biologically plausible and also suitable for neuromorphic hardware. We show in simulations that the algorithm can perform sparse feature extraction and manifold learning, two tasks which c…
A complex interplay of single-neuron properties and the recurrent network structure shapes the activity of cortical neurons. The single-neuron activity statistics differ in general from the respective population statistics, including spectra and, correspondingly, autocorrelation times. We develop a theory for self-consistent second-order single-neuron statistics in block-structured sparse random networks of spiking neurons. In particular, the theory predicts the neuron-level autocorrelation times, also known as intrinsic timescales, of the neuronal activity. The theory is based on an extension…
Spontaneous fluctuations and stimulus response are essential features of neural functioning but how they are connected is poorly understood. I derive fluctuation-dissipation relations (FDR) between the spontaneous spike and voltage correlations and the firing rate susceptibility for i) the leaky integrate-and-fire (IF) model with white noise; ii) an IF model with arbitrary voltage dependence, an adaptation current, and correlated noise. The FDRs can be used to derive correlation statistics or to infer the system's response from observations of its spontaneous activity.
The quadratic integrate-and-fire (QIF) model captures the normal form bifurcation dynamics of Type-I neurons found in cortex. Remarkably, this model is known to have a dual equivalent representation in terms of phase, called the $θ$-model, which has advantages for numerical simulations and analysis over the QIF model. Here, I investigate the nature of the dual representation and derive the general phase model expression for all integrate-and-fire models. Moreover, I show the condition for which the phase models exhibit discontinuous onset firing rate, the hallmark of Type-II spiking neurons.
Neural populations exposed to a certain stimulus learn to represent it better. However, the process that leads local, self-organized rules to do so is unclear. We address the question of how can a neural periodic input be learned and use the Differential Hebbian Learning framework, coupled with a homeostatic mechanism to derive two self-consistency equations that lead to increased responses to the same stimulus. Although all our simulations are done with simple Leaky-Integrate and Fire neurons and standard Spiking Time Dependent Plasticity learning rules, our results can be easily interpreted …
Maintaining the ability to fire sparsely is crucial for information encoding in neural networks. Additionally, spiking homeostasis is vital for spiking neural networks with changing numbers of weights and neurons. We discuss a range of network stabilisation approaches, inspired by homeostatic synaptic plasticity mechanisms reported in the brain. These include weight scaling, and weight change as a function of the network's spiking activity. We tested normalisation of the sum of weights for all neurons, and by neuron type. We examined how this approach affects firing rate and performance on clu…
We revisit the dynamics of a prototypical model of balanced activity in networks of spiking neutrons. A detailed investigation of the thermodynamic limit for fixed density of connections (massive coupling) shows that, when inhibition prevails, the asymptotic regime is not asynchronous but rather characterized by a self-sustained irregular, macroscopic (collective) dynamics. So long as the connectivity is massive, this regime is found in many different setups: leaky as well as quadratic integrate-and-fire neurons; large and small coupling strength; weak and strong external currents.
We have studied neuronal synchronisation in a random network of adaptive exponential integrate-and-fire neurons. We study how spiking or bursting synchronous behaviour appears as a function of the coupling strength and the probability of connections, by constructing parameter spaces that identify these synchronous behaviours from measurements of the inter-spike interval and the calculation of the order parameter. Moreover, we verify the robustness of synchronisaton by applying an external perturbation to each neuron. The simulations show that bursting synchronisation is more robust than spike …
Spike correlations between neurons are ubiquitous in the cortex, but their role is at present not understood. Here we describe the firing response of a leaky integrate-and-fire neuron (LIF) when it receives a temporarily correlated input generated by presynaptic correlated neuronal populations. Input correlations are characterized in terms of the firing rates, Fano factors, correlation coefficients and correlation timescale of the neurons driving the target neuron. We show that the sum of the presynaptic spike trains cannot be well described by a Poisson process. Solutions of the output firing…
A large network of integrate-and-fire neurons is studied analytically when the synaptic weights are independently randomly distributed according to a Gaussian distribution with arbitrary mean and variance. The relevant order parameters are identified, and it is shown that such network is statistically equivalent to an ensemble of independent integrate-and-fire neurons with each input signal given by the sum of a self-interaction deterministic term and a Gaussian colored noise. The model is able to reproduce the quasi-synchronous oscillations, and the dropout of their frequency, of the central …
Artificial neural networks normally require precise weights to operate, despite their origins in biological systems, which can be highly variable and noisy. When implementing artificial networks which utilize analog 'synaptic' devices to encode weights, however, inherent limits are placed on the accuracy and precision with which these values can be encoded. In this work, we investigate the effects that inaccurate synapses have on spiking neurons and spiking neural networks. Starting with a mathematical analysis of integrate-and-fire (IF) neurons, including different non-idealities (such as lea…
In spiking neural networks, the information is conveyed by the spike times, that depend on the intrinsic dynamics of each neuron, the input they receive and on the connections between neurons. In this article we study the Markovian nature of the sequence of spike times in stochastic neural networks, and in particular the ability to deduce from a spike train the next spike time, and therefore produce a description of the network activity only based on the spike times regardless of the membrane potential process. To study this question in a rigorous manner, we introduce and study an event-based …
Spiking Neural Networks (SNNs) have shown great potential in solving deep learning problems in an energy-efficient manner. However, they are still limited to simple classification tasks. In this paper, we propose Spiking-GAN, the first spike-based Generative Adversarial Network (GAN). It employs a kind of temporal coding scheme called time-to-first-spike coding. We train it using approximate backpropagation in the temporal domain. We use simple integrate-and-fire (IF) neurons with very high refractory period for our network which ensures a maximum of one spike per neuron. This makes the model …
We describe a new, computationally simple method for analyzing the dynamics of neuronal spike trains driven by external stimuli. The goal of our method is to test the predictions of simple spike-generating models against extracellularly recorded neuronal responses. Through a new statistic called the power ratio, we distinguish between two broad classes of responses: (1) responses that can be completely characterized by a variable firing rate, (for example, modulated Poisson and gamma spike trains); and (2) responses for which firing rate variations alone are not sufficient to characterize resp…
George Washington University（乔治华盛顿大学）；Nanyang Technological University（南洋理工大学）
The ability to rapidly identify symmetry and anti-symmetry is an essential attribute of intelligence. Symmetry perception is a central process in human vision and may be key to human 3D visualization. While previous work in understanding neuron symmetry perception has concentrated on the neuron as an integrator, here we show how the coincidence detecting property of the spiking neuron can be used to reveal symmetry density in spatial data. We develop a method for synchronizing symmetry-identifying spiking artificial neural networks to enable layering and feedback in the network. We show a meth…
A unique feature of neuromorphic computing is that memory is an implicit part of processing through traces of past information in the system's collective dynamics. The extent of memory about past inputs is commonly quantified by the autocorrelation time of collective dynamics. Based on past experimental evidence, a potential explanation for the underlying autocorrelations are close-to-critical fluctuations. Here, we show for self-organized networks of excitatory and inhibitory leaky integrate-and-fire neurons that autocorrelations can originate from emergent bistability upon reducing external …
We provide rigorous and exact results characterizing the statistics of spike trains in a network of leaky integrate and fire neurons, where time is discrete and where neurons are submitted to noise, without restriction on the synaptic weights. We show the existence and uniqueness of an invariant measure of Gibbs type and discuss its properties. We also discuss Markovian approximations and relate them to the approaches currently used in computational neuroscience to analyse experimental spike trains statistics.
The effect of a temporally correlated afferent current on the firing rate of a leaky integrate-and-fire (LIF) neuron is studied. This current is characterized in terms of rates, auto and cross-correlations, and correlation time scale $τ_c$ of excitatory and inhibitory inputs. The output rate $ν_{out}$ is calculated in the Fokker-Planck (FP) formalism in the limit of both small and large $τ_c$ compared to the membrane time constant $τ$ of the neuron. By simulations we check the analytical results, provide an interpolation valid for all $τ_c$ and study the neuron's response to rapid changes in t…
In machine learning, error back-propagation in multi-layer neural networks (deep learning) has been impressively successful in supervised and reinforcement learning tasks. As a model for learning in the brain, however, deep learning has long been regarded as implausible, since it relies in its basic form on a non-local plasticity rule. To overcome this problem, energy-based models with local contrastive Hebbian learning were proposed and tested on a classification task with networks of rate neurons. We extended this work by implementing and testing such a model with networks of leaky integrate…
Spatiotemporal patterns such as traveling waves are frequently observed in recordings of neural activity. The mechanisms underlying the generation of such patterns are largely unknown. Previous studies have investigated the existence and uniqueness of different types of waves or bumps of activity using neural-field models, phenomenological coarse-grained descriptions of neural-network dynamics. But it remains unclear how these insights can be transferred to more biologically realistic networks of spiking neurons, where individual neurons fire irregularly. Here, we employ mean-field theory to r…
University of Michigan（密歇根大学）；University of Western Australia（西澳大利亚大学）
Spiking neural networks can compensate for quantization error by encoding information either in the temporal domain, or by processing discretized quantities in hidden states of higher precision. In theory, a wide dynamic range state-space enables multiple binarized inputs to be accumulated together, thus improving the representational capacity of individual neurons. This may be achieved by increasing the firing threshold, but make it too high and sparse spike activity turns into no spike emission. In this paper, we propose the use of `threshold annealing' as a warm-up method for firing thresho…
The seemingly stochastic transient dynamics of neocortical circuits observed in vivo have been hypothesized to represent a signature of ongoing stochastic inference. In vitro neurons, on the other hand, exhibit a highly deterministic response to various types of stimulation. We show that an ensemble of deterministic leaky integrate-and-fire neurons embedded in a spiking noisy environment can attain the correct firing statistics in order to sample from a well-defined target distribution. We provide an analytical derivation of the activation function on the single cell level; for recurrent netwo…
How spiking networks are able to perform probabilistic inference is an intriguing question, not only for understanding information processing in the brain, but also for transferring these computational principles to neuromorphic silicon circuits. A number of computationally powerful spiking network models have been proposed, but most of them have only been tested, under ideal conditions, in software simulations. Any implementation in an analog, physical system, be it in vivo or in silico, will generally lead to distorted dynamics due to the physical properties of the underlying substrate. In t…
Spiking networks that perform probabilistic inference have been proposed both as models of cortical computation and as candidates for solving problems in machine learning. However, the evidence for spike-based computation being in any way superior to non-spiking alternatives remains scarce. We propose that short-term plasticity can provide spiking networks with distinct computational advantages compared to their classical counterparts. In this work, we use networks of leaky integrate-and-fire neurons that are trained to perform both discriminative and generative tasks in their forward and back…
Stanford University（斯坦福大学）；University of Oxford（牛津大学）
A vast majority of computation in the brain is performed by spiking neural networks. Despite the ubiquity of such spiking, we currently lack an understanding of how biological spiking neural circuits learn and compute in-vivo, as well as how we can instantiate such capabilities in artificial spiking circuits in-silico. Here we revisit the problem of supervised learning in temporally coding multi-layer spiking neural networks. First, by using a surrogate gradient approach, we derive SuperSpike, a nonlinear voltage-based three factor learning rule capable of training multi-layer networks of dete…
We introduce an exactly integrable version of the well-known leaky integrate-and-fire (LIF) model, with continuous membrane potential at the spiking event, the c-LIF. We investigate the dynamical regimes of a fully connected network of excitatory c-LIF neurons in the presence of short-term synaptic plasticity. By varying the coupling strength among neurons, we show that a complex chaotic dynamics arises, characterized by scale free avalanches. The origin of this phenomenon in the c-LIF can be related to the order symmetry breaking in neurons spike-times, which corresponds to the onset of a bro…
Over the past decade there has been a growing interest in the development of parallel hardware systems for simulating large-scale networks of spiking neurons. Compared to other highly-parallel systems, GPU-accelerated solutions have the advantage of a relatively low cost and a great versatility, thanks also to the possibility of using the CUDA-C/C++ programming languages. NeuronGPU is a GPU library for large-scale simulations of spiking neural network models, written in the C++ and CUDA-C++ programming languages, based on a novel spike-delivery algorithm. This library includes simple LIF (leak…
We aim at building a bridge between to {\it a priori} disconnected fields: Neuroscience and Material Science. We construct an analogy based on identifying spikes events in time with the positions of particles of matter. We show that one may think of the dynamical states of spiking neurons and spiking neural networks as {\it time-matter}. Namely, a structure of spike-events in time having analogue properties to that of ordinary matter. We can define for neural systems notions equivalent to the equations of state, phase diagrams and their phase transitions. For instance, the familiar Ideal Gas L…
