arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.08484cs.LG
收藏

FOSI:混合一阶与二阶优化

FOSI: Hybrid First and Second Order Optimization

Technion(以色列理工学院);York University(约克大学)

Hadar Sivan, Moshe Gabel, Assaf Schuster

AI总结:

针对高维下二阶优化曲率计算难的问题,提出元算法FOSI,通过正交子空间拆分结合一、二阶优化,在理论上分析了收敛性,实验中提升了一阶优化器性能且优于现有二阶方法。

摘要:

主流机器学习方法因高维场景下曲率计算难度大而放弃使用二阶信息。我们提出FOSI,这是一种新型元算法,通过在优化过程中高效融入二阶信息来提升任意基础一阶优化器的性能。每轮迭代中,FOSI隐式地将函数拆分为两个定义在正交子空间上的二次函数,随后采用二阶方法最小化第一个函数,用基础优化器最小化另一个。我们形式化分析了FOSI的收敛性以及它能提升基础优化器性能的条件。实验评估表明,FOSI提升了Heavy-Ball、Adam等一阶方法的收敛速度与优化耗时,且性能优于K-FAC和L-BFGS等二阶方法。

英文摘要:

Popular machine learning approaches forgo second-order information due to the difficulty of computing curvature in high dimensions. We present FOSI, a novel meta-algorithm that improves the performance of any base first-order optimizer by efficiently incorporating second-order information during the optimization process. In each iteration, FOSI implicitly splits the function into two quadratic functions defined on orthogonal subspaces, then uses a second-order method to minimize the first, and the base optimizer to minimize the other. We formally analyze FOSI's convergence and the conditions under which it improves a base optimizer. Our empirical evaluation demonstrates that FOSI improves the convergence rate and optimization time of first-order methods such as Heavy-Ball and Adam, and outperforms second-order methods (K-FAC and L-BFGS).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑