arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2101.11517cs.LGcs.CVmath.DSmath.OC
收藏

从统一的角度研究学习和视觉的双层优化:调查及其他

Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond

Dalian University of Technology(大连理工大学);Southern University of Science and Technology(南方科技大学);Xi’an Jiaotong University(西安交通大学);Peking University(北京大学);Shanghai Jiao Tong University(上海交通大学)

Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, Zhouchen Lin

摘要:

双层优化(BLO)是由经济博弈论发展而来,并被引入到优化领域的。BLO能够处理具有层次结构的问题,涉及两个级别的优化任务,其中一个任务嵌套在另一个任务中。在机器学习和计算机视觉领域,尽管动机和机制不同,但许多复杂问题,如超参数优化、多任务和元学习、神经结构搜索、对抗学习和深度强化学习,实际上都包含一系列密切相关的子问题。在本文中,我们首先从BLO的角度统一表达这些复杂的学习和视觉问题。然后,我们构建了一个基于最佳响应的单级重构,并建立了一个统一的算法框架,以理解和制定主流的基于梯度的BLO方法,涵盖的方面,从基本的自动微分计划,各种加速,简化,扩展及其收敛性和复杂性。最后,我们讨论了我们的统一BLO框架的潜力,设计新的算法,并指出一些有前途的方向,为未来的研究。

英文摘要:

Bi-Level Optimization (BLO) is originated from the area of economic game theory and then introduced into the optimization community. BLO is able to handle problems with a hierarchical structure, involving two levels of optimization tasks, where one task is nested inside the other. In machine learning and computer vision fields, despite the different motivations and mechanisms, a lot of complex problems, such as hyper-parameter optimization, multi-task and meta-learning, neural architecture search, adversarial learning and deep reinforcement learning, actually all contain a series of closely related subproblms. In this paper, we first uniformly express these complex learning and vision problems from the perspective of BLO. Then we construct a best-response-based single-level reformulation and establish a unified algorithmic framework to understand and formulate mainstream gradient-based BLO methodologies, covering aspects ranging from fundamental automatic differentiation schemes to various accelerations, simplifications, extensions and their convergence and complexity properties. Last but not least, we discuss the potentials of our unified BLO framework for designing new algorithms and point out some promising directions for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑