arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.12396math.OCcs.LG
收藏

有限和耦合组合随机优化:理论与应用

Finite-Sum Coupled Compositional Stochastic Optimization: Theory and Applications

The University of Iowa(爱荷华大学);Texas A&M University(德州农工大学)

Bokun Wang, Tianbao Yang

AI总结:

本文提出有限和耦合组合优化(FCCO)问题,针对非凸与凸目标给出带小批量移动平均估计的随机算法收敛分析,实现并行加速的改进预言复杂度,实验验证了理论结论。

摘要:

本文研究组合函数和的随机优化问题,其中每个求和项的内层函数与对应的求和索引耦合。我们将这类问题称为有限和耦合组合优化(finite-sum coupled compositional optimization, FCCO)。它在机器学习中有着广泛应用,可用于优化非凸或凸的组合测度/目标,例如平均精度(average precision, AP)、p-范数推动(p-norm push)、列表式排序损失、邻域成分分析(neighborhood component analysis, NCA)、深度生存分析、深度潜变量模型等,值得更精细的分析。然而,现有算法和分析在各方面存在局限。本文的贡献是针对非凸和凸目标,为一种简单的随机算法提供全面的收敛性分析。我们的核心结果是,通过使用带小批量的移动平均估计器,获得了具有并行加速的改进预言复杂度。我们的理论分析还通过对外层和内层采样等大小批次,为改进实际实现提供了新见解。在AP最大化、NCA和p-norm push上的数值实验验证了理论的部分结论。

英文摘要:

This paper studies stochastic optimization for a sum of compositional functions, where the inner-level function of each summand is coupled with the corresponding summation index. We refer to this family of problems as finite-sum coupled compositional optimization (FCCO). It has broad applications in machine learning for optimizing non-convex or convex compositional measures/objectives such as average precision (AP), p-norm push, listwise ranking losses, neighborhood component analysis (NCA), deep survival analysis, deep latent variable models, etc., which deserves finer analysis. Yet, existing algorithms and analyses are restricted in one or other aspects. The contribution of this paper is to provide a comprehensive convergence analysis of a simple stochastic algorithm for both non-convex and convex objectives. Our key result is the improved oracle complexity with the parallel speed-up by using the moving-average based estimator with mini-batching. Our theoretical analysis also exhibits new insights for improving the practical implementation by sampling the batches of equal size for the outer and inner levels. Numerical experiments on AP maximization, NCA, and p-norm push corroborate some aspects of the theory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑