In this paper, we introduce an unbiased gradient simulation algorithms for solving convex optimization problem with stochastic function compositions. We show that the unbiased gradient generated from the algorithm has finite variance and finite expected computation cost. We then combined the unbiased gradient simulation with two variance reduced algorithms (namely SVRG and SCSG) and showed that the proposed optimization algorithms based on unbiased gradient simulations exhibit satisfactory convergence properties. Specifically, in the SVRG case, the algorithm with simulated gradient can be shown to converge linearly to optima in expectation and almost surely under strong convexity. Finally, for the numerical experiment,we applied the algorithms to two important cases of stochastic function compositions optimization: maximizing the Cox's partial likelihood model and training conditional random fields.
Tight Heavy-Tailed Complexity Bounds for Optimizing Polyak--Łojasiewicz Objectives
Shanghai University of Finance and Economics(上海财经大学);MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics(上海财经大学计算与经济交叉研究教育部重点实验室)
Optimizing the Preconditioner: A Black-box Online-to-Nonconvex Conversion with Static Regret Minimization Oracles
Center for Computational Science and Engineering, MIT(麻省理工学院计算科学与工程中心);Department of Civil and Environmental Engineering, MIT(麻省理工学院土木与环境工程系);Institute for Data, Systems, and Society, MIT(麻省理工学院数据、系统与社会研究所)