Sample Complexity of Stochastic Optimization with Integer Variables
Dept. of Applied Mathematics & Statistics(应用数学与统计学系);Johns Hopkins University(约翰霍普金斯大学);Microsoft Research (Redmond)(微软研究院(红mond));Dept. of Computer Science, PUC-Rio(计算机科学系,巴西圣保罗大学)
Hongyu Cheng, Yinghao Zheng, Marco Molinaro, Amitabh Basu
We establish sample complexity results for stochastic optimization over the integers, especially with a view to understand the complexity with respect to the corresponding continuous optimization problem. We show that integer optimization can sometimes require strictly more samples and sometimes strictly smaller number of samples, depending on the structure of the objective and constraints. 1. For Lipschitz objectives over subsets of the $\ell_\infty$ ball, the statistical complexity of general stochastic mixed-integer, nonlinear, nonconvex optimization is exactly the same as stochastic linear optimization with just bound constraints. 2. For Lipschitz objectives over subsets of the $\ell_2$ ball, we show that integer optimization can require strictly *smaller* sample size compared to the continuous setting in a certain regime. To get to this result, we also establish tight sample complexity results for nonconvex continuous stochastic optimization which, to the best of our knowledge, do not appear in prior work. 3. For strongly convex, smooth objectives, integer optimization has high statistical complexity compared to the continuous setting. In particular, we show that integer optimization requires $Ω(1/ε^2)$ samples to report an $ε$-approximate solution, compared to the well-known $O(1/ε)$ sample complexity from the continuous optimization literature.
SPABA: A Single-Loop and Probabilistic Stochastic Bilevel Algorithm Achieving Optimal Sample Complexity
National Center for Applied Mathematics Shenzhen, Southern University of Science and Technology, Shenzhen, China(深圳应用数学中心,南方科技大学,深圳,中国);Institute of Operations Research and Information Engineering, Beijing University of Technology, Beijing 100124, P.R. China(运筹与信息工程研究所,北京理工大学,北京100124,中华人民共和国);Department of Mathematics, Southern University of Science and Technology, Shenzhen, China(南方科技大学数学系,深圳,中国)