训练离散神经网络的最大和算法
A Max-Sum algorithm for training discrete neural networks
摘要:
我们提出了一个有效的学习算法的问题,训练神经网络的离散突触,一个著名的硬(NP完全)离散优化问题。该算法是所谓的最大和(MS)算法的变体。特别是,我们展示了如何,有界整数权重与$q$不同的状态和独立的凹先验分布(如$l_{1}$正则化),该算法的时间复杂度可以扩展为$O\left(N\log N\right)$每个节点更新,从而把它与替代方案,如置信传播(BP),而不诉诸近似。有两种特殊情况特别令人感兴趣:二元突触$W\in\{-1,1\}$和具有$l_{0}$正则化的三元突触$W\in\{-1,0,1\}$。我们提出的算法在二进制感知器学习问题上表现得和BP一样好,并且可能更适合于解决完全连接的两层网络上的问题,因为使用MS方法自然地打破了两层网络中固有的对称性。
英文摘要:
We present an efficient learning algorithm for the problem of training neural networks with discrete synapses, a well-known hard (NP-complete) discrete optimization problem. The algorithm is a variant of the so-called Max-Sum (MS) algorithm. In particular, we show how, for bounded integer weights with $q$ distinct states and independent concave a priori distribution (e.g. $l_{1}$ regularization), the algorithm's time complexity can be made to scale as $O\left(N\log N\right)$ per node update, thus putting it on par with alternative schemes, such as Belief Propagation (BP), without resorting to approximations. Two special cases are of particular interest: binary synapses $W\in\{-1,1\}$ and ternary synapses $W\in\{-1,0,1\}$ with $l_{0}$ regularization. The algorithm we present performs as well as BP on binary perceptron learning problems, and may be better suited to address the problem on fully-connected two-layer networks, since inherent symmetries in two layer networks are naturally broken using the MS approach.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。