arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.11292math.OC
收藏

熵风险规避的广义动量方法

Entropic Risk-Averse Generalized Momentum Methods

Bugra Can, Mert Gürbüzbalaban

AI总结:

本文研究随机梯度噪声下动量方法的收敛速率与次优性风险权衡,提出熵风险规避的GMM和AGD方法,通过系统选择参数在风险与收敛速度间取得平衡,并在二次优化和逻辑回归上验证了性能提升。

摘要:

在受随机梯度噪声影响的一阶算法背景下,我们研究了收敛速率(用于量化初始条件被遗忘的速度)与次优性“风险”(即偏离期望次优性的程度)之间的权衡。我们关注一类广义动量方法(GMM),根据GMM参数的选择,该类方法可以恢复流行的方法,如梯度下降(GD)、加速梯度下降(AGD)和重球(HB)方法作为特例。我们使用著名的风险度量“熵风险”和“熵风险价值”来量化次优性风险。对于强凸光滑最小化问题,我们首先通过一个统一理论获得了GMM的新收敛速率结果,该理论同样适用于AGD和HB,并改进了HB的一些现有结果。然后,我们给出了在给定迭代点处次优性的熵风险和熵风险价值的显式界,这也基于Chernoff不等式直接给出了次优性超过给定阈值的概率界。我们的结果揭示了收敛速率与次优性风险之间的基本权衡。随后,我们将所获得的熵风险和收敛速率估计代入一个计算上可处理的优化框架中,提出了熵风险规避的GMM(RA-GMM)和熵风险规避的AGD(RA-AGD)方法,这些方法能够选择GMM参数以系统地在熵风险价值与收敛速率之间进行权衡。我们表明,与标准的参数选择相比,RA-AGD和RA-GMM在二次优化和逻辑回归问题上带来了改进的性能。据我们所知,我们的工作是首次以系统的方式借助相干风险度量来设计动量方法的参数。

英文摘要:

In the context of first-order algorithms subject to random gradient noise, we study the trade-offs between the convergence rate (which quantifies how fast the initial conditions are forgotten) and the "risk" of suboptimality, i.e. deviations from the expected suboptimality. We focus on a general class of momentum methods (GMM) which recover popular methods such as gradient descent (GD), accelerated gradient descent (AGD), and heavy-ball (HB) method as special cases depending on the choice of GMM parameters. We use well-known risk measures "entropic risk" and "entropic value at risk" to quantify the risk of suboptimality. For strongly convex smooth minimization, we first obtain new convergence rate results for GMM with a unified theory that is also applicable to both AGD and HB, improving some of the existing results for HB. We then provide explicit bounds on the entropic risk and entropic value at risk of suboptimality at a given iterate which also provides direct bounds on the probability that the suboptimality exceeds a given threshold based on Chernoff's inequality. Our results unveil fundamental trade-offs between the convergence rate and the risk of suboptimality. We then plug the entropic risk and convergence rate estimates we obtained in a computationally tractable optimization framework and propose entropic risk-averse GMM (RA-GMM) and entropic risk-averse AGD (RA-AGD) methods which can select the GMM parameters to systematically trade-off the entropic value at risk with the convergence rate. We show that RA-AGD and RA-GMM lead to improved performance on quadratic optimization and logistic regression problems compared to the standard choice of parameters. To our knowledge, our work is the first to resort to coherent measures to design the parameters of momentum methods in a systematic manner.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑