ISAAC Newton:用于牛顿法的基于输入的近似曲率
ISAAC Newton: Input-based Approximate Curvature for Newton's Method
AI总结:
本研究提出ISAAC牛顿法,通过仅基于层输入计算近似曲率调节梯度,在小批量场景下计算开销渐近可忽略,兼具一阶与二阶方法的训练竞争力。
摘要:
我们提出ISAAC(基于输入的近似曲率),这是一种利用选定的二阶信息对梯度进行调节的新方法,在假设批量大小小于神经元数量的前提下,其计算开销会渐近消失。我们证明,仅基于对应层的输入即可计算出良好的调节因子,且不会产生大量计算开销。该方法即使在小批量随机场景下也能实现有效训练,使其兼具一阶与二阶方法的竞争力。
英文摘要:
We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size smaller than the number of neurons. We show that it is possible to compute a good conditioner based on only the input to a respective layer without a substantial computational overhead. The proposed method allows effective training even in small-batch stochastic regimes, which makes it competitive to first-order as well as second-order methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。