二值化神经网络中的突触亚可塑性
Synaptic Metaplasticity in Binarized Neural Networks
摘要:
虽然深度神经网络在许多情况下都超过了人类的表现,但它们很容易出现灾难性遗忘:在训练新任务时,它们会迅速忘记以前学习过的任务。基于理想化任务的神经科学研究表明,在大脑中,突触通过根据其过去的历史调整其可塑性来克服这个问题。然而,这种“化生”行为不会直接转移到深层神经网络中以减轻灾难性遗忘。在这项工作中,我们将二值化神经网络(深度神经网络的低精度版本)使用的隐藏权重解释为化生变量,并修改其训练技术以减轻遗忘。基于这一想法,我们提出并通过实验证明,在多任务和流学习的情况下,一种训练技术,可以减少灾难性遗忘,而不需要以前提供的数据,也不需要数据集之间的正式边界,并且性能接近更主流的技术与任务边界。我们支持我们的方法与理论分析一个易于处理的任务。这项工作连接了计算神经科学和深度学习,为未来的嵌入式和神经形态系统提供了重要的资产,特别是当使用具有类似于超塑性物理学的新型纳米器件时。
英文摘要:
While deep neural networks have surpassed human performance in multiple situations, they are prone to catastrophic forgetting: upon training a new task, they rapidly forget previously learned ones. Neuroscience studies, based on idealized tasks, suggest that in the brain, synapses overcome this issue by adjusting their plasticity depending on their past history. However, such "metaplastic" behaviours do not transfer directly to mitigate catastrophic forgetting in deep neural networks. In this work, we interpret the hidden weights used by binarized neural networks, a low-precision version of deep neural networks, as metaplastic variables, and modify their training technique to alleviate forgetting. Building on this idea, we propose and demonstrate experimentally, in situations of multitask and stream learning, a training technique that reduces catastrophic forgetting without needing previously presented data, nor formal boundaries between datasets and with performance approaching more mainstream techniques with task boundaries. We support our approach with a theoretical analysis on a tractable task. This work bridges computational neuroscience and deep learning, and presents significant assets for future embedded and neuromorphic systems, especially when using novel nanodevices featuring physics analogous to metaplasticity.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。