全局工作空间基元的收缩特性
Contraction Properties of the Global Workspace Primitive
AI总结:
针对多区域递归神经网络(RNNs)研究领域,本文扩展了Kozachkov等人提出的可证明稳定的RNNs of RNNs架构,证明其全局工作空间模块化结构的宽松稳定性条件,并通过稀疏组合网络实证验证了低参数模型的性能与鲁棒性,还提升了基准任务上的最先进性能。
摘要:
为推进多区域递归神经网络(RNNs)这一重要新兴研究领域,我们从理论和实证两方面扩展了Kozachkov等人在《RNNs of RNNs:递归构建稳定递归神经网络集合》中提出的可证明稳定的RNNs of RNNs。我们为该架构的显著特殊情况证明了宽松的稳定性条件,最显著的是全局工作空间模块化结构的情况。随后,我们展示了具有少量可训练参数的全局工作空间稀疏组合网络(Global Workspace Sparse Combo Nets)的实证成功,不仅通过强大的整体测试性能,还通过对单个子网络移除更强的恢复力。全局工作空间区域间拓扑的这些实证结果取决于稳定性的保持,突显了我们理论工作对实现模块化RNN成功的相关性。此外,通过更广泛地探索不同子网络模块间连接结构的稀疏性,我们在基准序列处理任务上提升了稳定RNN的最先进性能,从而强调了专用图结构对多区域RNN的普遍效用。
英文摘要:
To push forward the important emerging research field surrounding multi-area recurrent neural networks (RNNs), we expand theoretically and empirically on the provably stable RNNs of RNNs introduced by Kozachkov et al. in "RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks". We prove relaxed stability conditions for salient special cases of this architecture, most notably for a global workspace modular structure. We then demonstrate empirical success for Global Workspace Sparse Combo Nets with a small number of trainable parameters, not only through strong overall test performance but also greater resilience to removal of individual subnetworks. These empirical results for the global workspace inter-area topology are contingent on stability preservation, highlighting the relevance of our theoretical work for enabling modular RNN success. Further, by exploring sparsity in the connectivity structure between different subnetwork modules more broadly, we improve the state of the art performance for stable RNNs on benchmark sequence processing tasks, thus underscoring the general utility of specialized graph structures for multi-area RNNs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。