次优势密集簇允许离散突触神经网络中的简单学习和高计算性能
Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
摘要:
我们表明,离散突触权重可以有效地用于大规模神经系统的学习,并导致意外的计算性能。我们专注于学习随机模式的单层网络中的二进制突触的代表性情况。标准的统计分析表明,这个问题是指数主导的孤立的解决方案,是非常难以找到算法。在这里,我们介绍了一种新的方法,使我们能够找到分析证据的次优势和极其密集的区域的解决方案的存在。数值实验证实了这些发现。我们还表明,密集的区域是令人惊讶的简单的学习协议,这些突触配置是强大的扰动和推广比典型的解决方案。这些结果扩展到具有多个状态的突触和更深层次的神经架构。大偏差度量还建议如何基于局部熵最大化设计新的优化算法方案。
英文摘要:
We show that discrete synaptic weights can be efficiently used for learning in large scale neural systems, and lead to unanticipated computational performance. We focus on the representative case of learning random patterns with binary synapses in single layer networks. The standard statistical analysis shows that this problem is exponentially dominated by isolated solutions that are extremely hard to find algorithmically. Here, we introduce a novel method that allows us to find analytical evidence for the existence of subdominant and extremely dense regions of solutions. Numerical experiments confirm these findings. We also show that the dense regions are surprisingly accessible by simple learning protocols, and that these synaptic configurations are robust to perturbations and generalize better than typical solutions. These outcomes extend to synapses with multiple states and to deeper neural architectures. The large deviation measure also suggests how to design novel algorithmic schemes for optimization based on local entropy maximization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。