We introduce a general framework for large-scale model-based derivative-free optimization based on iterative minimization within random subspaces. We present a probabilistic worst-case complexity analysis for our method, where in particular we prove high-probability bounds on the number of iterations before a given optimality is achieved. This framework is specialized to nonlinear least-squares problems, with a model-based framework based on the Gauss-Newton method. This method achieves scalability by constructing local linear interpolation models to approximate the Jacobian, and computes new steps at each iteration in a subspace with user-determined dimension. We then describe a practical implementation of this framework, which we call DFBGN. We outline efficient techniques for selecting the interpolation points and search subspace, yielding an implementation that has a low per-iteration linear algebra cost (linear in the problem dimension) while also achieving fast objective decrease as measured by evaluations. Extensive numerical results demonstrate that DFBGN has improved scalability, yielding strong performance on large-scale nonlinear least-squares problems.
Powell-Style Model-Based Derivative-Free Optimization with Complexity Guarantees
Department of Applied Mathematics, University of Colorado, Boulder（科罗拉多大学博尔德分校应用数学系）；School of Industrial and Systems Engineering, Georgia Tech（佐治亚理工学院工业与系统工程学院）
Scalable Derivative-Free Optimization for Nonlinear Least-Squares Problems
Mathematical Institute, University of Oxford（牛津大学数学研究所）；The Alan Turing Institute for Data Science（艾伦·图灵数据科学研究所）；Big Data Institute, University of Oxford（牛津大学大数据研究所）；Mathematical Sciences Institute, Australian National University（澳大利亚国立大学数学科学研究所）
