arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.04739stat.MLcs.LGmath.CT
收藏

机器学习中的非自然算法

Unnatural Algorithms in Machine Learning

Oden Institute for Computational Engineering and Sciences(奥登计算工程与科学研究所);University of Texas(得克萨斯大学)

Christian Goodbrake

AI总结:

本文用范畴论中的自然变换刻画优化算法对网络重参数化的不变性,提出可涵盖投影等不可逆映射的推广框架,并发现多数常用训练算法并不具备这种自然性。

摘要:

自然梯度下降具有一个显著性质:在小学习率极限下,它对网络重参数化表现出不变性,因此即使对于高度协变的网络参数化,也能产生稳健的训练行为。我们表明,具有这一性质的优化算法可被视为自然变换的离散近似:该自然变换从“依据优化器状态空间的位形流形上的微分同胚群来确定该状态空间”的函子,指向“依据该群来确定该状态空间切丛”的函子。具有此性质的算法在训练参数化不良的网络时效率更高,因为它们产生的网络演化近似地对网络重参数化保持不变。更具体地说,当学习率趋于零时,这些算法生成的流在光滑重参数化下不变,而各参数的相应流由等变映射确定。通过将该性质表述为自然变换,我们能够推广到超出群作用等变性的情形;该框架可以处理投影等不可逆映射,从而建立一个框架,用于直接比较非异构网络架构之间的训练行为,并在网络规模增大时通过考察这些投影的逆极限(如果存在)来形式化研究极限行为。我们介绍了一种更一般地引入这种自然性的简单方法,并考察了多种流行的机器学习训练算法,发现其中大多数都是非自然的。

英文摘要:

Natural gradient descent has a remarkable property that in the small learning rate limit, it displays an invariance with respect to network reparameterizations, leading to robust training behavior even for highly covariant network parameterizations. We show that optimization algorithms with this property can be viewed as discrete approximations of natural transformations from the functor determining an optimizer's state space from the diffeomorphism group if its configuration manifold, to the functor determining that state space's tangent bundle from this group. Algorithms with this property enjoy greater efficiency when used to train poorly parameterized networks, as the network evolution they generate is approximately invariant to network reparameterizations. More specifically, the flow generated by these algorithms in the limit as the learning rate vanishes is invariant under smooth reparameterizations, the respective flows of the parameters being determined by equivariant maps. By casting this property a natural transformation, we allow for generalizations beyond equivariance with respect to group actions; this framework can account for non-invertible maps such as projections, creating a framework for the direct comparison of training behavior across non-isomorphic network architectures, and the formal examination of limiting behavior as network size increases by considering inverse limits of these projections, should they exist. We introduce a simple method of introducing this naturality more generally and examine a number of popular machine learning training algorithms, finding that most are unnatural.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑