arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.14813cs.LG
收藏

模范数中的可扩展优化

Scalable Optimization in the Modular Norm

Columbia University(哥伦比亚大学);Lawrence Livermore National Lab(劳伦斯利弗莫尔国家实验室);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室)

Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, Jeremy Bernstein

AI总结:

本文提出模范数,一种递归定义的神经网络权重空间自然范数,用于归一化优化器更新以实现学习率跨宽度和深度转移,并证明梯度Lipschitz连续性,提供Modula包实现。

摘要:

为了提升当代深度学习的性能,人们有兴趣在层数和层大小两方面扩展神经网络。当增加单层宽度时,训练的优雅扩展与在该层特定“自然范数”中归一化权重及其更新相关联。在本文中,我们通过定义模范数(modular norm)显著推广了这一思想,模范数是任何神经网络架构的完整权重空间上的自然范数。模范数与网络架构本身递归地共同定义。我们展示了模范数具有几个有前景的应用。在实际方面,模范数可用于归一化任何基础优化器的更新,使得学习率在宽度和深度上可转移。这意味着用户无需计算优化器特定的缩放因子即可扩展训练。在理论方面,我们表明对于任何由“行为良好”的原子模块构建的神经网络,网络的梯度在模范数下是Lipschitz连续的,且Lipschitz常数具有简单的递归公式。这一表征为将优化理论中的标准思想移植到深度学习打开了大门。我们创建了一个名为Modula的Python包,它自动在架构的模范数中归一化权重更新。该包可通过“pip install modula”获取,源代码位于https://github.com/jxbz/modula。

英文摘要:

To improve performance in contemporary deep learning, one is interested in scaling up the neural network in terms of both the number and the size of the layers. When ramping up the width of a single layer, graceful scaling of training has been linked to the need to normalize the weights and their updates in the "natural norm" particular to that layer. In this paper, we significantly generalize this idea by defining the modular norm, which is the natural norm on the full weight space of any neural network architecture. The modular norm is defined recursively in tandem with the network architecture itself. We show that the modular norm has several promising applications. On the practical side, the modular norm can be used to normalize the updates of any base optimizer so that the learning rate becomes transferable across width and depth. This means that the user does not need to compute optimizer-specific scale factors in order to scale training. On the theoretical side, we show that for any neural network built from "well-behaved" atomic modules, the gradient of the network is Lipschitz-continuous in the modular norm, with the Lipschitz constant admitting a simple recursive formula. This characterization opens the door to porting standard ideas in optimization theory over to deep learning. We have created a Python package called Modula that automatically normalizes weight updates in the modular norm of the architecture. The package is available via "pip install modula" with source code at https://github.com/jxbz/modula.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑