具有最优神谕复杂度的复合在线到非凸转换
Composite Online-to-Nonconvex Conversion with Optimal Oracle Complexity
浏览论文内容
中文总结 AI 辅助
针对随机非光滑非凸复合优化问题，扩展在线到非凸转换框架，提出新的在线学习器损失，得到的算法在一阶或零阶查询下达到最优神谕复杂度，且凸正则项未增加复杂度。
中文摘要 AI 辅助
我们研究随机非光滑非凸复合优化问题，该问题包含约束优化、神经网络正则化训练等多个重要问题。目标函数是一个可能非光滑的非凸Lipschitz函数与一个凸正则项的和，且该函数通过随机梯度或函数值进行访问。我们的目标是找到满足复合目标的Goldstein型平稳性条件的点。据我们所知，在一阶访问下，该场景尚无已知的神谕复杂度界，而现有零阶访问下的复杂度是次优的。为解决这一问题，我们采用在线到非凸转换框架，该框架通过在线学习器选择更新方向，且已知其对非复合问题可达到最优速率。我们将该框架扩展到复合场景，为学习器引入新的损失函数，这些损失函数包含正则项本身而非其线性化，且在线镜像下降的变体可实现低遗憾。我们证明，所得算法可在$O(\delta^{-1}\varepsilon^{-3})$次随机梯度查询或$O(d\delta^{-1}\varepsilon^{-3})$次函数值查询下找到此类点，其中$\delta$为Goldstein半径，$\varepsilon$为平稳性容差，$d$为维度。这些速率与非复合非光滑非凸优化的最优速率匹配，表明额外的凸正则项不会使神谕复杂度恶化。我们还给出了光滑情形的速率并呈现了数值实验。
英文摘要
We consider stochastic nonsmooth nonconvex composite optimization, which includes several important problems such as constrained optimization and the regularized training of neural networks. The objective is the sum of a possibly nonsmooth nonconvex Lipschitz function and a convex regularizer, and the function is accessed through stochastic gradients or function values. The goal is to find a point that satisfies a Goldstein-type stationarity condition designed for composite objectives. To our knowledge, no oracle complexity bound for this setting is known under first-order access, and existing complexities under zeroth-order access are suboptimal. To handle this issue, we employ the framework of online-to-nonconvex conversion, which chooses update directions by an online learner and is known to achieve optimal rates for noncomposite problems. We extend the framework to our composite scenario by introducing new losses for the learner, which contain the regularizer itself rather than its linearization and for which a variant of online mirror descent achieves low regret. We show that the resulting algorithm finds such a point with $O(δ^{-1}\varepsilon^{-3})$ stochastic gradient queries or $O(dδ^{-1}\varepsilon^{-3})$ function-value queries, where $δ$ is the Goldstein radius, $\varepsilon$ is the stationarity tolerance, and $d$ is the dimension. These rates match the optimal ones for noncomposite nonsmooth nonconvex optimization, demonstrating that the additional convex regularizer does not worsen the oracle complexity. We also give rates for the smooth case and present numerical experiments.
发表机构
- The University of Tokyo（东京大学）
- The University of Osaka（大阪大学）
- RIKEN（理化学研究所）
机构由 AI 辅助整理，请以论文原文为准。
