arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.04569math.OC
收藏

自适应方差缩减的统一理论

Unified Theory of Adaptive Variance Reduction

Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)(人工智能基础研究所(BRAIn Lab))

Aleksandr Shestakov, Valery Parfenov, Aleksandr Beznosikov

AI总结:

本文提出自适应方差缩减的统一理论,将有偏估计纳入统一框架,并设计无需超参数调优的自适应步长方差缩减方法,覆盖有限和、分布式与坐标优化问题。

摘要:

方差缩减是一类强大的随机优化机制,在许多机器学习任务中表现出有效性。它基于利用某些递归序列估计精确梯度。此前,许多论文证明了具有无偏方差缩减估计量的方法可以在单一框架中描述。我们推广了这一方法,并表明无偏性假设是多余的;因此,我们将有偏估计量纳入该分析。但我们工作的主要贡献是提出了新的方差缩减方法,这些方法具有在算法迭代过程中自适应调整的步长,并且不需要超参数调优。我们的分析涵盖有限和问题、分布式优化和坐标方法。在各种任务中的数值实验验证了我们方法的有效性。

英文摘要:

Variance reduction is a family of powerful mechanisms for stochastic optimization that appears to be helpful in many machine learning tasks. It is based on estimating the exact gradient with some recursive sequences. Previously, many papers demonstrated that methods with unbiased variance-reduction estimators can be described in a single framework. We generalize this approach and show that the unbiasedness assumption is excessive; hence, we include biased estimators in this analysis. But the main contribution of our work is the proposition of new variance reduction methods with adaptive step sizes that are adjusted throughout the algorithm iterations and, moreover, do not need hyperparameter tuning. Our analysis covers finite- sum problems, distributed optimization, and coordinate methods. Numerical experiments in various tasks validate the effectiveness of our methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑