理解递归神经网络的原理:解决模型复杂性的生成方法
Understanding the Principles of Recursive Neural networks: A Generative Approach to Tackle Model Complexity
摘要:
递归神经网络是一种能够学习深层结构化信息的非线性自适应模型。然而,这些模式尚未被广泛接受。这主要是由于其内在的复杂性。特别地,不仅因为是极其复杂的信息处理模型,而且因为计算昂贵的学习阶段。这些模型最流行的训练方法是通过结构的反向传播。由于收敛问题,该算法已被证明不是最适合于结构化处理,而更复杂的训练方法以显著增加计算成本为代价来提高收敛速度。在本文中,我们首先分析了这些模型背后的基本原理,旨在了解它们的计算能力。其次,提出了一种近似二阶随机学习算法。所提出的算法动态地适应整个网络的训练阶段的学习率,而不会产生过于昂贵的计算工作量。该算法运行在在线和批处理模式。此外,所得到的学习计划是强大的对消失梯度问题。该算法的优点与现实世界的应用实例证明。
英文摘要:
Recursive Neural Networks are non-linear adaptive models that are able to learn deep structured information. However, these models have not yet been broadly accepted. This fact is mainly due to its inherent complexity. In particular, not only for being extremely complex information processing models, but also because of a computational expensive learning phase. The most popular training method for these models is back-propagation through the structure. This algorithm has been revealed not to be the most appropriate for structured processing due to problems of convergence, while more sophisticated training methods enhance the speed of convergence at the expense of increasing significantly the computational cost. In this paper, we firstly perform an analysis of the underlying principles behind these models aimed at understanding their computational power. Secondly, we propose an approximate second order stochastic learning algorithm. The proposed algorithm dynamically adapts the learning rate throughout the training phase of the network without incurring excessively expensive computational effort. The algorithm operates in both on-line and batch modes. Furthermore, the resulting learning scheme is robust against the vanishing gradients problem. The advantages of the proposed algorithm are demonstrated with a real-world application example.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。