Variance reduction techniques such as SPIDER/SARAH/STORM have been extensively studied to improve the convergence rates of stochastic non-convex optimization, which usually maintain and update a sequence of estimators for a single function across iterations. What if we need to track multiple functional mappings across iterations but only with access to stochastic samples of $\mathcal{O}(1)$ functional mappings at each iteration? There is an important application in solving an emerging family of coupled compositional optimization problems in the form of $\sum_{i=1}^m f_i(g_i(\mathbf{w}))$, where $g_i$ is accessible through a stochastic oracle. The key issue is to track and estimate a sequence of $\mathbf g(\mathbf{w})=(g_1(\mathbf{w}), \ldots, g_m(\mathbf{w}))$ across iterations, where $\mathbf g(\mathbf{w})$ has $m$ blocks and it is only allowed to probe $\mathcal{O}(1)$ blocks to attain their stochastic values and Jacobians. To improve the complexity for solving these problems, we propose a novel stochastic method named Multi-block-Single-probe Variance Reduced (MSVR) estimator to track the sequence of $\mathbf g(\mathbf{w})$. It is inspired by STORM but introduces a customized error correction term to alleviate the noise not only in stochastic samples for the selected blocks but also in those blocks that are not sampled. With the help of the MSVR estimator, we develop several algorithms for solving the aforementioned compositional problems with improved complexities across a spectrum of settings with non-convex/convex/strongly convex/Polyak-Łojasiewicz (PL) objectives. Our results improve upon prior ones in several aspects, including the order of sample complexities and dependence on the strong convexity parameter. Empirical studies on multi-task deep AUC maximization demonstrate the better performance of using the new estimator.
Compositional Stochastic Average Gradient for Machine Learning and Related Applications
The Pennsylvania State University(宾夕法尼亚州立大学);College of Information Science and Technology(信息科学与技术学院);Department of Computer Science and Engineering(计算机科学与工程系)
Unbiased estimators for the variance of MMD estimators
最大平均差异(MMD)是在许多应用中有用的概率分布之间的基于核的距离(Gretton et al. 2012),具有令人满意的计算和统计特性的简单估计量。能够有效地估计该估计量的方差,对解决双样本检验中的各种问题都有很大的帮助。为此,Bounliphone等人。(2016)使用U-统计量理论来推导MMD估计量方差的估计量,以及两个这样的估计量之间的差异。然而,他们的估计器丢弃了低阶项,并且不必要地存在偏差。我们在这篇笔记中表明-扩展和纠正Sutherland等人的工作。(2017)-我们可以找到一个真正的无偏估计量,用于平方MMD估计量和两个相关平方MMD估计量之差的实际方差,基本上没有额外的计算成本。
Explicit construction of the minimum error variance estimator for stochastic LTI state-space systems
Dept. of Electronic Systems, Aalborg University, Aalborg, Denmark(丹麦奥尔堡大学电子系统系);Laboratoire Signal et Automatique de Lille (CRIStAL), Lille, France(法国里尔信号与自动化实验室(CRIStAL))
VarGrad: A Low-Variance Gradient Estimator for Variational Inference
Freie Universität Berlin(柏林自由大学);BTU Cottbus-Senftenberg(科特布斯-森夫滕贝格工业大学);dida Datenschmiede GmbH(迪达数据工坊有限公司);University of Warwick(华威大学);Universität Potsdam(波茨坦大学);DeepMind(深度思维);The Alan Turing Institute(艾伦·图灵研究所)
Gradient Estimation for Binary Latent Variables via Gradient Variance Clipping
Columbia University(哥伦比亚大学);Memorial Sloan Kettering Cancer Center(纪念斯隆-凯特琳癌症中心);Howard Hughes Medical Institute(霍华德·休斯医学研究所);University of Florida(佛罗里达大学)