arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1308.6003cs.NEcs.ITmath.IT
收藏

通过逃避虚假不动点来改善稀疏联想记忆

Improving Sparse Associative Memories by Escaping from Bogus Fixed Points

Zhe Yao, Vincent Gripon, Michael Rabbat

摘要:

Gripon-Berrou神经网络(GBNN)是最近发明的递归神经网络,其包含类似LDPC的稀疏编码设置,这使得其对噪声和错误具有极强的弹性。GBNN的一个自然用途是作为关联存储器。神经元动力学有两种激活规则,即总和和最大值之和。后者在检索率方面大大优于前者。在之前的讨论和实验中,人们相信,虽然总和可能导致网络振荡,但最大值和总是收敛到与先前存储的模式相对应的神经元团的集合。然而,这并不完全正确。事实上,最大值和经常收敛到虚假的不动点,其中集合只包括收敛状态的一个小子集。通过利用这个被忽视的事实,我们可以大大提高检索率。我们讨论了这个特殊的问题,并提出了一些算法,以推动最大值和超越这些虚假的不动点。为了直接和全面地解决这个问题,还开发了一种新的后处理算法,并根据GBNN的结构进行了定制。实验结果表明,新算法在检索率和运行时间方面都取得了巨大的提升,与标准的sum-of-max和所有其他算法相比。

英文摘要:

The Gripon-Berrou neural network (GBNN) is a recently invented recurrent neural network embracing a LDPC-like sparse encoding setup which makes it extremely resilient to noise and errors. A natural use of GBNN is as an associative memory. There are two activation rules for the neuron dynamics, namely sum-of-sum and sum-of-max. The latter outperforms the former in terms of retrieval rate by a huge margin. In prior discussions and experiments, it is believed that although sum-of-sum may lead the network to oscillate, sum-of-max always converges to an ensemble of neuron cliques corresponding to previously stored patterns. However, this is not entirely correct. In fact, sum-of-max often converges to bogus fixed points where the ensemble only comprises a small subset of the converged state. By taking advantage of this overlooked fact, we can greatly improve the retrieval rate. We discuss this particular issue and propose a number of heuristics to push sum-of-max beyond these bogus fixed points. To tackle the problem directly and completely, a novel post-processing algorithm is also developed and customized to the structure of GBNN. Experimental results show that the new algorithm achieves a huge performance boost in terms of both retrieval rate and run-time, compared to the standard sum-of-max and all the other heuristics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑