arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.06675cs.LGcs.AIcs.CLcs.CVcs.NE
收藏

优化算法的符号发现

Symbolic Discovery of Optimization Algorithms

Google(谷歌);UCLA(加州大学洛杉矶分校)

Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, Quoc V. Le

AI总结:

本文提出将算法发现视为程序搜索,发现优化算法Lion,它仅跟踪动量,通过符号运算更新,在多项任务上超越Adam,节省计算资源,并已应用于生产系统。

摘要:

我们提出了一种将算法发现表述为程序搜索的方法,并将其应用于发现用于深度神经网络训练的优化算法。我们利用高效的搜索技术来探索一个无限且稀疏的程序空间。为了弥合代理任务与目标任务之间巨大的泛化差距,我们还引入了程序选择与简化策略。我们的方法发现了一种简单而有效的优化算法,即Lion(进化符号动量)。它比Adam更节省内存,因为它只跟踪动量。与自适应优化器不同,其更新通过符号运算对每个参数具有相同的幅度。我们将Lion与广泛使用的优化器(如Adam和Adafactor)进行比较,用于在不同任务上训练各种模型。在图像分类方面,Lion在ImageNet上将ViT的准确率提升了高达2%,并在JFT上节省了高达5倍的预训练计算量。在视觉-语言对比学习中,我们在ImageNet上实现了88.3%的零样本准确率和91.1%的微调准确率,分别超越了之前的最佳结果2%和0.1%。在扩散模型上,Lion通过获得更好的FID分数并将训练计算量减少高达2.3倍,从而优于Adam。对于自回归、掩码语言建模和微调,Lion表现出与Adam相似或更好的性能。我们对Lion的分析表明,其性能提升随训练批次大小的增加而增长。由于符号函数产生的更新范数较大,它还需要比Adam更小的学习率。此外,我们考察了Lion的局限性,并确定了其改进较小或统计上不显著的情景。Lion还成功部署在生产系统中,如Google搜索广告点击率模型。

英文摘要:

We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, $\textbf{Lion}$ ($\textit{Evo$\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}$tum}$). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3% $\textit{zero-shot}$ and 91.1% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant. Lion is also successfully deployed in production systems such as Google search ads CTR model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑