发表机构
Nanjing University of Science and Technology; Nanjing University（南京理工大学; 南京大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对有限和耦合组合优化问题，提出多块单探针方差缩减估计器，在部分块采样下高效追踪多函数，改进非凸等目标的样本复杂度，并在多任务深度AUC最大化中验证优越性。
AI 中文摘要
传统的方差缩减方法（如SPIDER、SARAH、STORM）已被广泛研究，用于提高随机优化的收敛速度。这些技术通常在迭代过程中为单个函数（或梯度）维护一系列估计器。然而，如果我们需要跟踪多个函数，但每次迭代只能访问$\mathcal{O}(1)$个函数的随机样本，该怎么办？这种场景出现在一类重要的新兴有限和耦合组合优化（FCCO）问题中，其形式为$\frac{1}{m}\sum_{i=1}^m f_i(g_i(\mathbf{w}))$，其中每个$g_i$只能通过随机预言机访问。关键挑战在于随时间跟踪$\mathbf g(\mathbf{w})=(g_1(\mathbf{w}), \ldots, g_m(\mathbf{w}))$，其中$\mathbf g(\mathbf{w})$有$m$个块，但每一步只能探测其中$\mathcal{O}(1)$个块的随机值。为应对这一挑战，我们提出了一种新颖的多块单探针方差缩减（MSVR）估计器，以在部分块采样下高效追踪$\mathbf g(\mathbf{w})$。基于MSVR估计器，我们为FCCO问题开发了多种算法，在非凸、凸、强凸和Polyak-Łojasiewicz（PL）目标上实现了改进的样本复杂度。当外部函数梯度$\nabla f_i$为线性时，我们进一步获得了对$m$的改进依赖。在多任务深度AUC最大化上的实证研究进一步证明了所提估计器的优越性能。
英文摘要
Traditional variance reduction methods (e.g., SPIDER, SARAH, STORM) have been extensively investigated for improving the convergence rates of stochastic optimization. These techniques typically maintain a sequence of estimators for a single function (or gradient) across iterations. However, what if we need to track multiple functions, but can only access stochastic samples of $\mathcal{O}(1)$ functions at each iteration? This scenario arises in an important emerging family of finite-sum coupled compositional optimization (FCCO) problems of the form $\frac{1}{m}\sum_{i=1}^m f_i(g_i(\mathbf{w}))$, where each $g_i$ is accessible only through a stochastic oracle. The key challenge is to track $\mathbf g(\mathbf{w})=(g_1(\mathbf{w}), \ldots, g_m(\mathbf{w}))$ over time, where $\mathbf g(\mathbf{w})$ has $m$ blocks but only $\mathcal{O}(1)$ blocks can be probed for their stochastic values at each step. To address this challenge, we propose a novel Multi-block-Single-probe Variance Reduction (MSVR) estimator to efficiently trace $\mathbf g(\mathbf{w})$ under partial block sampling. Building on the MSVR estimator, we develop several algorithms for FCCO problems, achieving improved sample complexities for non-convex, convex, strongly convex, and Polyak-Łojasiewicz (PL) objectives. We further obtain an improved dependence on $m$ when the outer function gradients $\nabla f_i$ are linear. Empirical studies on multi-task deep AUC maximization further demonstrate the superior performance of the proposed estimators.
