arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.00604cs.LGcs.CVmath.OCstat.ML
收藏

ISAAC Newton:用于牛顿法的基于输入的近似曲率

ISAAC Newton: Input-based Approximate Curvature for Newton's Method

University of Konstanz(康斯坦茨大学);University of Salzburg(萨尔茨堡大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);University of Michigan(密歇根大学);Stanford University(斯坦福大学);University of Frankfurt(法兰克福大学)

Felix Petersen, Tobias Sutter, Christian Borgelt, Dongsung Huh, Hilde Kuehne, Yuekai Sun, Oliver Deussen

AI总结:

本研究提出ISAAC牛顿法,通过仅基于层输入计算近似曲率调节梯度,在小批量场景下计算开销渐近可忽略,兼具一阶与二阶方法的训练竞争力。

摘要:

我们提出ISAAC(基于输入的近似曲率),这是一种利用选定的二阶信息对梯度进行调节的新方法,在假设批量大小小于神经元数量的前提下,其计算开销会渐近消失。我们证明,仅基于对应层的输入即可计算出良好的调节因子,且不会产生大量计算开销。该方法即使在小批量随机场景下也能实现有效训练,使其兼具一阶与二阶方法的竞争力。

英文摘要:

We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size smaller than the number of neurons. We show that it is possible to compute a good conditioner based on only the input to a respective layer without a substantial computational overhead. The proposed method allows effective training even in small-batch stochastic regimes, which makes it competitive to first-order as well as second-order methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑