arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1606.06069cs.LG
收藏

学习大型复杂模型的相对自然梯度算法

Relative Natural Gradient for Learning Large Complex Models

École Polytechnique(巴黎综合理工学院);Sony CSL(索尼计算机科学实验室)

Ke Sun, Frank Nielsen

摘要:

Fisher信息和自然梯度为人工神经网络提供了深刻的见解和有力的工具。然而,随着学习者结构的庞大和复杂,相关分析变得越来越困难。这篇论文朝着一个新的方向迈出了初步的一步。我们提取一个大的神经元系统的局部组件,并定义其相对Fisher信息度量,准确地描述这个小组件,是不变的系统的其他部分。这个概念是重要的,因为几何结构大大简化,它可以很容易地应用于指导神经网络的学习。我们提供了一个常用组件的列表上的分析,并演示了如何使用这个概念,以进一步提高优化。

英文摘要:

Fisher information and natural gradient provided deep insights and powerful tools to artificial neural networks. However related analysis becomes more and more difficult as the learner's structure turns large and complex. This paper makes a preliminary step towards a new direction. We extract a local component of a large neuron system, and defines its relative Fisher information metric that describes accurately this small component, and is invariant to the other parts of the system. This concept is important because the geometry structure is much simplified and it can be easily applied to guide the learning of neural networks. We provide an analysis on a list of commonly used components, and demonstrate how to use this concept to further improve optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑