Gradient-based first-order adaptive optimization methods such as the Adam optimizer are prevalent in training artificial networks, achieving the state-of-the-art results. This work attempts to answer the question whether it is viable for biological neural systems to adopt such optimization methods. To this end, we demonstrate a realization of the Adam optimizer using biologically-plausible mechanisms in synapses. The proposed learning rule has clear biological correspondence, runs continuously in time, and achieves performance to comparable Adam's. In addition, we present a new approach, inspired by the predisposition property of synapses observed in neuroscience, to circumvent the biological implausibility of the weight transport problem in backpropagation (BP). With only local information and no separate training phases, this method establishes and maintains weight symmetry in the forward and backward signaling paths, and is applicable to the proposed biologically plausible Adam learning rule. These mechanisms may shed light on the way in which biological synaptic dynamics facilitate learning.
Adaptive Sparse Structure Development with Pruning and Regeneration for Spiking Neural Networks
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences(中国科学院脑科学与智能技术卓越创新中心)
Adaptive structure evolution and biologically plausible synaptic plasticity for recurrent spiking neural networks
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences(中国科学院脑科学与智能技术卓越创新中心)
Temporal-adaptive Weight Quantization for Spiking Neural Networks
Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算机学院);Harbin Institute of Technology(哈尔滨工业大学);Pengcheng Laboratory(鹏城实验室);Institute of Computing and Intelligence (ICI), Harbin Institute of Technology(哈尔滨工业大学计算机与智能信息学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Distributed Recurrent Neural Forward Models with Synaptic Adaptation for Complex Behaviors of Walking Robots
Institute for Physics - Biophysics, George-Agust-University(乔治-奥古斯特大学物理-生物物理研究所);Bernstein Center for Computational Neuroscience, George-Agust-University(乔治-奥古斯特大学伯恩斯坦计算神经科学中心);Riken Brain Science Institute(理化学研究所脑科学研究所);Maersk Mc-Kinney Moller Institute, University of Southern Denmark(南丹麦大学马士基·麦金尼·穆勒研究所)
MorphSNN: Adaptive Graph Diffusion and Structural Plasticity for Spiking Neural Networks
School of Software, Northeastern University(东北大学软件学院);Guangdong Institute of Intelligence Science and Technolog(广东智能科学技术研究院);The Hong Kong Polytechnic University(香港理工大学);Monash University(蒙纳士大学)