由连接性和功能约束实现的稀疏学习
Sparse learning enabled by constraints on connectivity and function
AI总结:
本文利用可精确求解的联想学习模型,比较不同稀疏性约束,发现 $l_0$ 范数给出最优稀疏度,而剪除弱连接可近乎同等高效且支持在线实施。
摘要:
稀疏连接是大脑的一个标志性特征,也是人工神经网络所期望的属性。它能够提升能源效率、简化训练并增强网络功能的鲁棒性。因此,详细理解如何在不损害网络性能的情况下实现稀疏性,对于神经科学、深度学习和神经形态计算应用均有益处。我们采用一个可精确求解的联想学习模型,评估了各种诱导稀疏性的约束对连接性和功能的影响。我们确定了由 $l_0$ 范数约束实现的最优稀疏水平,并发现通过消除弱连接几乎可以获得相同的效率。我们表明,这种实现稀疏性的方法可以在线实施,从而使其与神经科学和机器学习应用相兼容。
英文摘要:
Sparse connectivity is a hallmark of the brain and a desired property of artificial neural networks. It promotes energy efficiency, simplifies training, and enhances the robustness of network function. Thus, a detailed understanding of how to achieve sparsity without jeopardizing network performance is beneficial for neuroscience, deep learning, and neuromorphic computing applications. We used an exactly solvable model of associative learning to evaluate the effects of various sparsity-inducing constraints on connectivity and function. We determine the optimal level of sparsity achieved by the $l_0$ norm constraint and find that nearly the same efficiency can be obtained by eliminating weak connections. We show that this method of achieving sparsity can be implemented online, making it compatible with neuroscience and machine learning applications.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。