arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1906.11684cs.NEcs.LGstat.ML
收藏

谐振器网络在解决高维向量分解方面优于优化方法

Resonator Networks outperform optimization methods at solving high-dimensional vector factorization

University of California, Berkeley(加州大学伯克利分校);Intel Laboratories(英特尔实验室)

Spencer J. Kent, E. Paxon Frady, Friedrich T. Sommer, Bruno A. Olshausen

摘要:

我们发展了谐振器网络的理论基础,谐振器网络是Frady等人(2020)引入的一种新型递归神经网络,用于解决矢量符号架构中出现的高维矢量分解问题。给定由离散的一组高维向量之间的Hadamard乘积形成的复合向量,谐振器网络可以有效地将复合分解为这些因子。我们将谐振器网络的性能与基于优化的方法(包括交替最小二乘法和几种基于梯度的算法)进行了比较,结果表明谐振器网络在几个重要方面具有优势。这一优势是通过利用非线性动力学和“叠加搜索”的组合来实现的,通过该组合,正确解的估计是从所有可能解的加权叠加中形成的。虽然替代方法也在叠加中搜索,但谐振器网络的动态使它们能够在探索解空间和利用本地信息之间取得更有效的平衡,以推动网络走向可能的解。谐振器网络不能保证收敛,但在特定的制度下,他们几乎总是这样做。作为放松这种全局收敛保证的交换,谐振器网络在寻找因子分解方面比考虑的所有替代方法都更有效。

英文摘要:

We develop theoretical foundations of Resonator Networks, a new type of recurrent neural network introduced in Frady et al. (2020) to solve a high-dimensional vector factorization problem arising in Vector Symbolic Architectures. Given a composite vector formed by the Hadamard product between a discrete set of high-dimensional vectors, a Resonator Network can efficiently decompose the composite into these factors. We compare the performance of Resonator Networks against optimization-based methods, including Alternating Least Squares and several gradient-based algorithms, showing that Resonator Networks are superior in several important ways. This advantage is achieved by leveraging a combination of nonlinear dynamics and "searching in superposition," by which estimates of the correct solution are formed from a weighted superposition of all possible solutions. While the alternative methods also search in superposition, the dynamics of Resonator Networks allow them to strike a more effective balance between exploring the solution space and exploiting local information to drive the network toward probable solutions. Resonator Networks are not guaranteed to converge, but within a particular regime they almost always do. In exchange for relaxing this guarantee of global convergence, Resonator Networks are dramatically more effective at finding factorizations than all alternative approaches considered.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑