Universal methods for optimization are designed to achieve theoretically optimal convergence rates without any prior knowledge of the problem's regularity parameters or the accurarcy of the gradient oracle employed by the optimizer. In this regard, existing state-of-the-art algorithms achieve an $\mathcal{O}(1/T^2)$ value convergence rate in Lipschitz smooth problems with a perfect gradient oracle, and an $\mathcal{O}(1/\sqrt{T})$ convergence rate when the underlying problem is non-smooth and/or the gradient oracle is stochastic. On the downside, these methods do not take into account the problem's dimensionality, and this can have a catastrophic impact on the achieved convergence rate, in both theory and practice. Our paper aims to bridge this gap by providing a scalable universal gradient method - dubbed UnderGrad - whose oracle complexity is almost dimension-free in problems with a favorable geometry (like the simplex, linearly constrained semidefinite programs and combinatorial bandits), while retaining the order-optimal dependence on $T$ described above. These "best-of-both-worlds" results are achieved via a primal-dual update scheme inspired by the dual exploration method for variational inequalities.
Almost Sure Convergence of Stochastic Approximation: An Interplay of Noise and Step Size
University of Information Technology, Vietnam National University（越南国立大学信息技术大学）；Institute of Operations Research and Analytics, National University of Singapore（新加坡国立大学运筹学与分析研究所）；H. Milton Stewart School of Industrial & Systems Engineering, Georgia Institute of Technology（佐治亚理工学院H.米尔顿·斯图尔特工业与系统工程学院）
