arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01627math.OC
收藏

为何自适应零阶优化有效?

Why Does Adaptive Zeroth-Order Optimization Work?

Xi’an Jiaotong University(西安交通大学);Fudan University(复旦大学)

Haishan Ye, Luo Luo

AI总结:

本研究揭示了自适应零阶优化方法中实证标准差与梯度范数的关系,从而在广义(L0,L1)-光滑性条件下建立了更高效的收敛速率和查询复杂度界限。

摘要:

零阶(ZO)优化在现实应用中流行,因为获取梯度信息可能昂贵或不可用。最近,自适应ZO方法通过函数值的实证标准差归一化梯度估计器,在实际性能上表现出色,特别是在微调大语言模型时。然而,这种策略的理论理解仍然有限。在本工作中,我们证明实证标准差在高概率下与梯度的范数紧密成比例。基于这一见解,我们分析了在广义(L0,L1)-光滑性条件下,相对于矩阵范数的自适应ZO方法。我们为确定性和随机设置建立了显式的收敛速率和查询复杂度界限,证明自适应ZO方法在收敛速度和查询效率上优于固定步长的普通ZO方法。

英文摘要:

Zeroth-order (ZO) optimization is popular in real-world applications that accessing the gradient information is expensive or unavailable. Recently, adaptive ZO methods that normalize gradient estimators by the empirical standard deviation of function values have achieved strong practical performance, particularly in fine-tuning the large language model. However, the theoretical understanding of such strategy remains limited. In this work, we show that the empirical standard deviation is, with high probability, closely proportional to the norm of the (stochastic) gradient. Based on this insight, we analyze adaptive ZO methods under the generalized $(L_0,L_1)$-smoothness condition with respect to the matrix norm. We establish explicit convergence rates and query complexity bounds for both deterministic and stochastic settings, demonstrating that adaptive ZO methods achieve the faster convergence and the improved query efficiency compared to the vanilla ZO methods with fixed-step.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑