arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2207.08540cs.LGmath.OC
收藏

组合优化问题的多块单探针方差缩减估计

Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimization

Nanjing University(南京大学);the University of Iowa(爱荷华大学);Texas A&M University(得克萨斯农工大学)

Wei Jiang, Gang Li, Yibo Wang, Lijun Zhang, Tianbao Yang

摘要:

方差减少技术,如SPIDER/SARAH/STORM已被广泛研究,以提高收敛速度的随机非凸优化,通常保持和更新一个序列的估计一个单一的功能跨越迭代。如果我们需要在迭代中跟踪多个函数映射,但只能在每次迭代中访问$\mathcal{O}(1)$函数映射的随机样本,该怎么办?有一个重要的应用,在解决一个新兴的家庭的耦合组合优化问题的形式$\sum_{i=1}^m f_i(g_i(\mathbf{w}))$,其中$g_i$是通过一个随机预言机访问。关键问题是跟踪和估计跨迭代的$\mathbf g(\mathbf{w})=(g_1(\mathbf{w}), \ldots, g_m(\mathbf{w}))$序列,其中$\mathbf g(\mathbf{w})$具有$m$块,并且仅允许探测$\mathcal{O}(1)$块以获得它们的随机值和雅可比矩阵。为了提高解决这些问题的复杂性,我们提出了一种新的随机方法称为多块单探针方差缩减(MSVR)估计跟踪序列的$\mathbf g(\mathbf{w})$。它受到STORM的启发,但引入了定制的纠错项,以减轻不仅在所选块的随机样本中的噪声,而且在未采样的那些块中的噪声。在MSVR估计器的帮助下,我们开发了几种算法,用于解决上述组合问题,并在非凸/凸/强凸/Polyak-Jasojasiewicz(PL)目标的设置范围内提高了复杂性。我们的结果在几个方面,包括样本复杂性的顺序和强凸性参数的依赖性,以前的改进。多任务深度AUC最大化的实证研究表明,使用新的估计更好的性能。

英文摘要:

Variance reduction techniques such as SPIDER/SARAH/STORM have been extensively studied to improve the convergence rates of stochastic non-convex optimization, which usually maintain and update a sequence of estimators for a single function across iterations. What if we need to track multiple functional mappings across iterations but only with access to stochastic samples of $\mathcal{O}(1)$ functional mappings at each iteration? There is an important application in solving an emerging family of coupled compositional optimization problems in the form of $\sum_{i=1}^m f_i(g_i(\mathbf{w}))$, where $g_i$ is accessible through a stochastic oracle. The key issue is to track and estimate a sequence of $\mathbf g(\mathbf{w})=(g_1(\mathbf{w}), \ldots, g_m(\mathbf{w}))$ across iterations, where $\mathbf g(\mathbf{w})$ has $m$ blocks and it is only allowed to probe $\mathcal{O}(1)$ blocks to attain their stochastic values and Jacobians. To improve the complexity for solving these problems, we propose a novel stochastic method named Multi-block-Single-probe Variance Reduced (MSVR) estimator to track the sequence of $\mathbf g(\mathbf{w})$. It is inspired by STORM but introduces a customized error correction term to alleviate the noise not only in stochastic samples for the selected blocks but also in those blocks that are not sampled. With the help of the MSVR estimator, we develop several algorithms for solving the aforementioned compositional problems with improved complexities across a spectrum of settings with non-convex/convex/strongly convex/Polyak-Łojasiewicz (PL) objectives. Our results improve upon prior ones in several aspects, including the order of sample complexities and dependence on the strong convexity parameter. Empirical studies on multi-task deep AUC maximization demonstrate the better performance of using the new estimator.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑