arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.13575cs.LGcs.AIcs.CVstat.ML
收藏

高效的非参数优化器搜索多种任务

Efficient Non-Parametric Optimizer Search for Diverse Tasks

Ruochen Wang, Yuanhao Xiong, Minhao Cheng, Cho-Jui Hsieh

摘要:

优化器的高效和自动化设计在全栈AutoML系统中起着至关重要的作用。然而,优化器搜索中的现有方法通常受到其可伸缩性、可生成性或样本效率的限制。为了使优化器搜索的研究和应用民主化,我们提出了第一个有效的,可扩展的和可推广的框架,可以直接搜索感兴趣的任务。我们首先观察到优化器更新基本上是应用于梯度的数学表达式。受底层数学表达式固有的树结构的启发,我们将优化器的空间重新排列成一棵超树,其中每条路径编码一个优化器。这样,优化器搜索就可以自然地表述为路径查找问题,从而允许使用各种成熟的树遍历方法作为搜索算法。我们采用了一个适应的蒙特卡罗方法树搜索,配备拒绝采样和等价形式检测,利用优化器更新规则的特点,以进一步提高样本效率。我们提供了一组不同的任务来对我们的算法进行基准测试,并证明,只有128个评估,所提出的框架可以发现超越人类设计的同行和以前的优化器搜索方法的优化器。

英文摘要:

Efficient and automated design of optimizers plays a crucial role in full-stack AutoML systems. However, prior methods in optimizer search are often limited by their scalability, generability, or sample efficiency. With the goal of democratizing research and application of optimizer search, we present the first efficient, scalable and generalizable framework that can directly search on the tasks of interest. We first observe that optimizer updates are fundamentally mathematical expressions applied to the gradient. Inspired by the innate tree structure of the underlying math expressions, we re-arrange the space of optimizers into a super-tree, where each path encodes an optimizer. This way, optimizer search can be naturally formulated as a path-finding problem, allowing a variety of well-established tree traversal methods to be used as the search algorithm. We adopt an adaptation of the Monte Carlo method to tree search, equipped with rejection sampling and equivalent-form detection that leverage the characteristics of optimizer update rules to further boost the sample efficiency. We provide a diverse set of tasks to benchmark our algorithm and demonstrate that, with only 128 evaluations, the proposed framework can discover optimizers that surpass both human-designed counterparts and prior optimizer search methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑