Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks

为平衡做贡献，按规则布线：从简单、生物合理的神经可塑性规则中涌现反向传播

Contribute to balance, wire in accordance: Emergence of backpropagation from a simple, bio-plausible neuroplasticity rule

针对反向传播（BP）的生物合理性争议，提出一种基于E-I平衡和逆行信号、在三个时间尺度运行的神经可塑性规则，数学证明其可精确复制分层网络BP，模拟显示能诱导不同群落结构，为BP的生物实现提供框架。

基于时间的反向传播在无单突触限制的脉冲神经网络中

Timing-Based Backpropagation in Spiking Neural Networks Without Single-Spike Restrictions

本文提出了一种新型反向传播算法，用于训练无单突触限制的脉冲神经网络，通过多个突触时间的相对关系提高计算能力，并在准确性上超越了传统模型。

Spike time displacement based error backpropagation in convolutional spiking neural networks

We recently proposed the STiDi-BP algorithm, which avoids backward recursive gradient computation, for training multi-layer spiking neural networks (SNNs) with single-spike-based temporal coding. The algorithm employs a linear approximation to compute the derivative of the spike latency with respect to the membrane potential and it uses spiking neurons with piecewise linear postsynaptic potential to reduce the computational cost and the complexity of neural processing. In this paper, we extend the STiDi-BP algorithm to employ it in deeper and convolutional architectures. The evaluation results…

Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagation

The timing of individual neuronal spikes is essential for biological brains to make fast responses to sensory stimuli. However, conventional artificial neural networks lack the intrinsic temporal coding ability present in biological networks. We propose a spiking neural network model that encodes information in the relative timing of individual neuron spikes. In classification tasks, the output of the network is indicated by the first neuron to spike in the output layer. This temporal coding scheme allows the supervised training of the network with backpropagation, using locally exact derivati…

Neural Network with Unbounded Activation Functions is Universal Approximator

This paper presents an investigation of the approximation property of neural networks with unbounded activation functions, such as the rectified linear unit (ReLU), which is the new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorkin distributions. By showing three reconstruction formulas by using the Fourier slice theorem, the Radon transform, and Parseval's relation, it is shown that a neural network with unbounded activation functions still satisfies the universal approximation property. As an additional consequence, the ri…

Tensor Switching Networks

We present a novel neural network algorithm, the Tensor Switching (TS) network, which generalizes the Rectified Linear Unit (ReLU) nonlinearity to tensor-valued hidden units. The TS network copies its entire input vector to different locations in an expanded representation, with the location determined by its hidden unit activity. In this way, even a simple linear readout from the TS representation can implement a highly expressive deep-network-like function. The TS network hence avoids the vanishing gradient problem by construction, at the cost of larger representation size. We develop severa…