arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2203.06474cs.LG
收藏

优化器合并

Optimizer Amalgamation

University of Texas at Austin(德克萨斯大学奥斯汀分校);Carnegie Mellon University(卡内基梅隆大学);Michigan State University(密歇根州立大学);University of California, Santa Barbara(加州大学圣巴巴拉分校);IBM Research(IBM研究院)

Tianshu Huang, Tianlong Chen, Sijia Liu, Shiyu Chang, Lisa Amini, Zhangyang Wang

摘要:

为给定的问题选择合适的优化器是研究人员和实践者的主要兴趣。许多分析优化已经提出了使用各种理论和经验的方法,但是,没有一个可以提供一个普遍的优势,比其他竞争性的优化。因此,我们的动机是研究一个新的问题,名为优化合并:我们如何才能最好地结合到一个单一的“学生”优化器,可以有更强的问题特定的性能的“老师”优化池?在本文中,我们从“学习优化”领域中得到启示,使用一个可学习的融合目标。首先,我们定义了三个可微合并机制合并池的分析优化梯度下降。然后,为了减少融合过程的方差,我们还探索了通过扰动融合目标来稳定融合过程的方法。最后,我们提出的实验表明,我们的合并优化器相比,其合并组件和学习优化基线的优越性,以及我们的方差减少扰动的功效。我们的代码和预训练模型可在http://github.com/VITA-Group/OptimizerAmalgamation上公开获取。

英文摘要:

Selecting an appropriate optimizer for a given problem is of major interest for researchers and practitioners. Many analytical optimizers have been proposed using a variety of theoretical and empirical approaches; however, none can offer a universal advantage over other competitive optimizers. We are thus motivated to study a new problem named Optimizer Amalgamation: how can we best combine a pool of "teacher" optimizers into a single "student" optimizer that can have stronger problem-specific performance? In this paper, we draw inspiration from the field of "learning to optimize" to use a learnable amalgamation target. First, we define three differentiable amalgamation mechanisms to amalgamate a pool of analytical optimizers by gradient descent. Then, in order to reduce variance of the amalgamation process, we also explore methods to stabilize the amalgamation process by perturbing the amalgamation target. Finally, we present experiments showing the superiority of our amalgamated optimizer compared to its amalgamated components and learning to optimize baselines, and the efficacy of our variance reducing perturbations. Our code and pre-trained models are publicly available at http://github.com/VITA-Group/OptimizerAmalgamation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑