arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.07976cs.LGcs.CVmath.DSmath.OC
收藏

一个基于遗传算法的双层优化的广义下降聚集框架

A General Descent Aggregation Framework for Gradient-based Bi-level Optimization

Dalian University of Technology(大连理工大学);Zhejiang University of Technology(浙江工业大学);The University of Hong Kong(香港大学);University of Victoria(维多利亚大学);Southern University of Science and Technology(南方科技大学)

Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, Jin Zhang

摘要:

近年来,在机器学习和计算机视觉领域,已经开发了各种基于梯度的方法来解决双层优化(BLO)问题。然而,这些现有方法的理论正确性和实际有效性总是依赖于一些限制条件(例如,LLS),这在现实世界的应用中很难得到满足。此外,以往的文献只证明了理论结果的基础上,其特定的迭代策略,从而缺乏一个通用的配方来统一分析不同的基于梯度的BLO的收敛行为。在这项工作中,我们制定从一个乐观的双层的观点,并建立一个新的基于梯度的算法框架,命名为双层下降聚合(BDA),部分解决上述问题。具体来说,BDA提供了一个模块化的结构,分层聚合的上层和下层的子问题,以产生我们的双层迭代动力学。在理论上,我们建立了一个通用的收敛性分析模板,并推导了一个新的证明配方,以研究基于梯度的BLO方法的基本理论性质。此外,这项工作系统地探讨了BDA在不同优化场景下的收敛行为,即,考虑到各种溶液质量(即,全局/局部/静态解)。大量的实验证明了我们的理论结果,并证明了所提出的算法的超参数优化和元学习任务的优越性。源代码可在https://github.com/vis-opt-group/BDA上获得。

英文摘要:

In recent years, a variety of gradient-based methods have been developed to solve Bi-Level Optimization (BLO) problems in machine learning and computer vision areas. However, the theoretical correctness and practical effectiveness of these existing approaches always rely on some restrictive conditions (e.g., Lower-Level Singleton, LLS), which could hardly be satisfied in real-world applications. Moreover, previous literature only proves theoretical results based on their specific iteration strategies, thus lack a general recipe to uniformly analyze the convergence behaviors of different gradient-based BLOs. In this work, we formulate BLOs from an optimistic bi-level viewpoint and establish a new gradient-based algorithmic framework, named Bi-level Descent Aggregation (BDA), to partially address the above issues. Specifically, BDA provides a modularized structure to hierarchically aggregate both the upper- and lower-level subproblems to generate our bi-level iterative dynamics. Theoretically, we establish a general convergence analysis template and derive a new proof recipe to investigate the essential theoretical properties of gradient-based BLO methods. Furthermore, this work systematically explores the convergence behavior of BDA in different optimization scenarios, i.e., considering various solution qualities (i.e., global/local/stationary solution) returned from solving approximation subproblems. Extensive experiments justify our theoretical results and demonstrate the superiority of the proposed algorithm for hyper-parameter optimization and meta-learning tasks. Source code is available at https://github.com/vis-opt-group/BDA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑