竞争性可塑性以降低学习的能耗
Competitive plasticity to reduce the energetic costs of learning
AI总结:
本文探讨了通过限制神经网络可塑性来减少学习能耗的方法,提出两种算法在降低能耗的同时仅带来微小的学习时间增加。
摘要:
大脑不仅受到计算所需能量的限制,还受到形成记忆所需能量的限制。实验显示,学习简单的条件任务已带来显著的代谢成本。然而,学习MNIST任务达到95%准确率似乎需要至少10^8次突触更新。因此,大脑可能进化出以最小能量学习的能力。我们探讨了馈送前向神经网络学习所需的能量。基于一个简洁的能量模型,我们提出两种限制可塑性的算法:1)只修改更新大的突触;2)限制可塑性到形成网络路径的突触子集。结合这两种方法可显著节省能量,同时仅带来微小的学习时间增加。在生物网络中,网络往往远大于任务需求。特别是在此情况下,可以实现较大的节省。因此,竞争性限制可塑性有助于节省与突触可塑性相关的代谢能量。这些结果可能有助于更好地理解生物可塑性,并使人工学习更贴近生物学习。此外,这些算法也可能对硬件有益,因为在电子学中,内存存储也是能耗较高的。
英文摘要:
The brain is not only constrained by energy needed to fuel computation, but it is also constrained by energy needed to form memories. Experiments have shown that learning simple conditioning tasks already carries a significant metabolic cost. Yet, learning a task like MNIST to 95% accuracy appears to require at least 10^{8} synaptic updates. Therefore the brain has likely evolved to be able to learn using as little energy as possible. We explored the energy required for learning in feedforward neural networks. Based on a parsimonious energy model, we propose two plasticity restricting algorithms that save energy: 1) only modify synapses with large updates, and 2) restrict plasticity to subsets of synapses that form a path through the network. Combining these two methods leads to substantial energy savings while only incurring a small increase in learning time. In biology networks are often much larger than the task requires. In particular in that case, large savings can be achieved. Thus competitively restricting plasticity helps to save metabolic energy associated to synaptic plasticity. The results might lead to a better understanding of biological plasticity and a better match between artificial and biological learning. Moreover, the algorithms might also benefit hardware because in electronics memory storage is energetically costly as well.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。