Introduction to Model-Based Derivative-Free Optimization
University of Melbourne（墨尔本大学）；School of Mathematics and Statistics（数学与统计学院）；ARC Training Centre in Optimisation Technologies, Integrated Methodologies, and Applications (OPTIMA)（ARC优化技术、集成方法论及应用训练中心）
The field of derivative-free optimization (DFO) studies algorithms for nonlinear optimization that do not rely on the availability of gradient or Hessian information. It is primarily designed for settings when functions are black-box, expensive to evaluate and/or noisy. A widely used and studied class of DFO methods for local optimization is model-based DFO, where the general principles from derivative-based nonlinear optimization algorithms are followed, but local Taylor-type approximations are replaced with alternative local models constructed by interpolation. This document provides an overview of the basic algorithms and analysis for model-based DFO, covering worst-case complexity, approximation theory for polynomial interpolation models, and extensions to constrained and noisy problems.
A novel class of derivative-free optimization algorithms is developed. The main idea is to utilize certain non-commutative maps in order to approximate the gradient of the objective function. Convergence properties of the novel algorithms are established and simulation examples are presented.
In this work, we propose an efficient method for solving box constrained derivative free optimization problems involving high dimensions. The proposed method relies on exploring the feasible region using a direct search approach based on scaled conjugate gradient with quadratic interpolation models. The extensive numerical computations carried out over test problems with varying dimensions demonstrate the performance of the proposed method for derivative free optimization.
A novel derivative-free algorithm, optimization by moving ridge functions (OMoRF), for unconstrained and bound-constrained optimization is presented. This algorithm couples trust region methodologies with output-based dimension reduction to accelerate convergence of model-based optimization strategies. The dimension-reducing subspace is updated as the trust region moves through the function domain, allowing OMoRF to be applied to functions with no known global low-dimensional structure. Furthermore, its low computational requirement allows it to make rapid progress when optimizing high-dimensi…
In this work we introduce the stochastic nonlinear constrained derivative-free optimization method (S)NOWPAC (Stochastic Nonlinear Optimization With Path-Augmented Constraints). The method extends the derivative-free optimizer NOWPAC to be applicable for optimization under uncertainty. It is based on a trust-region framework, utilizing local fully quadratic surrogate models combined with Gaussian process surrogates to mitigate the noise in the objective function and constraint evaluations. We show the performance of our algorithm on a variety of robust optimization problems from the CUTEst ben…
Derivatives are an important tool for single-objective optimization. In fact, it is commonly accepted that derivative-based methods present a better performance than derivative-free optimization approaches. In this work, we will show that the same does not apply to multiobjective derivative-based optimization, when the goal is to compute an approximation to the complete Pareto front of a given problem. The competitiveness of Direct MultiSearch (DMS), a robust and efficient derivative-free optimization algorithm, will be stated for derivative-based multiobjective optimization problems. We will …
A new universal derivative-free optimization method CDOS (Conjugate Direction with Orthogonal Shift) is proposed. The CDOS method was specially developed to solve optimization tasks where the objective function and constraints are black boxes. The method has quadratic convergence for quadratic and near quadratic functions. An objective function can be non-differentiable and non-continuous. For constrained optimization the constraints can also be non-differentiable and non-continuous. The method handles inequality constraints directly, i.e., it does not transform the objective function, nor use…
An adaptive regularization algorithm for unconstrained nonconvex optimization is proposed that is capable of handling inexact objective-function and derivative values, and also of providing approximate minimizer of arbitrary order. In comparison with a similar algorithm proposed in Cartis, Gould, Toint (2021), its distinguishing feature is that it is based on controlling the relative error between the model and objective values. A sharp evaluation complexity complexity bound is derived for the new algorithm.
In this article, we study the notion of gH-Hadamard derivative for interval-valued functions (IVFs) and its applications to interval optimization problems (IOPs). It is shown that the existence of gH-Hadamard derivative implies the existence of gH-Frechet derivative and vise-versa. Further, it is proved that the existence of gH-Hadamard derivative implies the existence of gH-continuity of IVFs. We found that the composition of a Hadamard differentiable real-valued function and a gH-Hadamard differentiable IVF is gHHadamard differentiable. Further, for finite comparable IVF, we prove that the g…
Redundancy is related to the amount of functionality that the structure can sustain in the worst-case scenario of structural degradation. This paper proposes a widely-applicable concept of redundancy optimization of finite-dimensional structures. The concept is consistent with the robust structural optimization, as well as the quantitative measure of structural redundancy based on the information-gap theory. A derivative-free algorithm is proposed based on the sequential quadratic programming (SQP) method, where we use the finite-difference method with adaptively varying the difference increme…
SOLNP+ is a derivative-free solver for constrained nonlinear optimization. It starts from SOLNP proposed in 1989 by Ye Ye with the main idea that uses finite difference to approximate the gradient. We incorporate the techniques of implicit filtering, new restart mechanism and modern quadratic programming solver into this new version with an ANSI C implementation. The algorithm exhibits a great advantage in running time and robustness under noise compared with the last version by MATLAB. SOLNP+ is free to download at https://github.com/COPT-Public/SOLNP_plus.
This work investigates finite differences and the use of interpolation models to obtain approximations to the first and second derivatives of a function. Here, it is shown that if a particular set of points is used in the interpolation model, then the solution to the associated linear system (i.e., approximations to the gradient and diagonal of the Hessian) can be obtained in $\mathcal{O}(n)$ computations, which is the same cost as finite differences, and is a saving over the $\mathcal{O}(n^3)$ cost when solving a general unstructured linear system. Moreover, if the interpolation points are fo…
In this work, multi-variable derivative-free optimization algorithms for unconstrained optimization problems are developed. A novel procedure for approximating the gradient of multi-variable objective functions based on non-commutative maps is introduced. The procedure is based on the construction of an exploration sequence to specify where the objective function is evaluated and the definition of so-called gradient generating functions which are composed with the objective function such that the procedure mimics a gradient descent algorithm. Various theoretical properties of the proposed clas…
We propose STARS, a randomized derivative-free algorithm for unconstrained optimization when the function evaluations are contaminated with random noise. STARS takes dynamic, noise-adjusted smoothing step-sizes that minimize the least-squares error between the true directional derivative of a noisy function and its finite difference approximation. We provide a convergence rate analysis of STARS for solving convex problems with additive or multiplicative noise. Experimental results show that (1) STARS exhibits noise-invariant behavior with respect to different levels of stochastic noise; (2) th…
In this paper we propose an heuristic to improve the performances of the recently proposed derivative-free method for nonsmooth optimization CS-DFN. The heuristic is based on a clustering-type technique to compute a direction { which relies on an estimate of Clarke's generalized gradient} of the objective function. As such, this direction (as it is shown by the numerical experiments) is a good descent direction for the objective function. We report some numerical results and comparison with the original CS-DFN method to show the utility of the proposed improvement on a set of well-known test p…
In many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize…
A regularization algorithm allowing random noise in derivatives and inexact function values is proposed for computing approximate local critical points of any order for smooth unconstrained optimization problems. For an objective function with Lipschitz continuous $p$-th derivative and given an arbitrary optimality order $q \leq p$, it is shown that this algorithm will, in expectation, compute such a point in at most $O\left(\left(\min_{j\in\{1,\ldots,q\}}ε_j\right)^{-\frac{p+1}{p-q+1}}\right)$ inexact evaluations of $f$ and its derivatives whenever $q\in\{1,2\}$, where $ε_j$ is the tolerance …
Direct search methods represent a robust and reliable class of algorithms for solving black-box optimization problems. In this paper, we explore the application of those strategies to Riemannian optimization, wherein minimization is to be performed with respect to variables restricted to lie on a manifold. More specifically, we consider classic and line search extrapolated variants of direct search, and, by making use of retractions, we devise tailored strategies for the minimization of both smooth and nonsmooth functions. As such we analyze, for the first time in the literature, a class of re…
In this paper, we consider mixed-integer nonsmooth constrained optimization problems whose objective/constraint functions are available only as the output of a black-box zeroth-order oracle (i.e., an oracle that does not provide derivative information) and we propose a new derivative-free linesearch-based algorithmic framework to suitably handle those problems. We first describe a scheme for bound constrained problems that combines a dense sequence of directions (to handle the nonsmoothness of the objective function) with primitive directions (to handle discrete variables). Then, we embed an e…
This paper studies the $H^1$ Sobolev seminorm of quadratic functions. The research is motivated by the least-norm interpolation that is widely used in derivative-free optimization. We express the $H^1$ seminorm of a quadratic function explicitly in terms of the Hessian and the gradient when the underlying domain is a ball. The seminorm gives new insights into least-norm interpolation. It clarifies the analytical and geometrical meaning of the objective function in least-norm interpolation. We employ the seminorm to study the extended symmetric Broyden update proposed by Powell. Numerical resul…
Derivative-Free optimization (DFO) focuses on designing methods to solve optimization problems without the analytical knowledge of gradients of the objective function. There are two main families of DFO methods: model-based methods and direct search methods. In model-based DFO methods, a model of the objective function is constructed using only objective function values, and the model is used to guide the computation of the next iterate. Natural questions in this class of algorithms include how many function evaluations should be used to construct the model? And, should this number be fixed, o…
Deformations of the computational mesh arising from optimization routines usually lead to decrease of mesh quality or even destruction of the mesh. We propose a theoretical framework using pre-shapes to generalize classical shape optimization and calculus. We define pre-shape derivatives and derive according structure and calculus theorems. In particular, tangential directions are featured in pre-shape derivatives, in contrast to classical shape derivatives featuring only normal directions. Techniques from classical shape optimization and -calculus are shown to carry over to this framework. An…
In this paper, we propose the StepDIRECT algorithm for derivative-free optimization (DFO), in which the black-box objective function has a stepwise landscape. Our framework is based on the well-known DIRECT algorithm. By incorporating the local variability to explore the flatness, we provide a new criterion to select the potentially optimal hyper-rectangles. In addition, we introduce a stochastic local search algorithm performing on potentially optimal hyper-rectangles to improve the solution quality and convergence speed. Global convergence of the StepDIRECT algorithm is provided. Numerical e…
We present a model-based derivative-free method for optimization subject to general convex constraints, which we assume are unrelaxable and accessed only through a projection operator that is cheap to evaluate. We prove global convergence and a worst-case complexity of $O(ε^{-2})$ iterations and objective evaluations for nonconvex functions, matching results for the unconstrained case. We introduce new, weaker requirements on model accuracy compared to existing methods. As a result, sufficiently accurate interpolation models can be constructed only using feasible points. We develop a comprehen…
A piecewise constant Mayer cost function is used to model optimal control problems in which the state space is partitioned into several regions, each having its own Mayer cost value. In such a context, the standard numerical methods used in optimal control theory naturally fail, due to the discontinuities and the null gradients associated with the Mayer cost function. In this paper an hybrid numerical method, based on both derivative-free optimization and smooth optimization techniques, is proposed to solve this class of problems. Numerical simulations are performed on some standard control sy…
A class of non-smooth and non-convex optimization problems with penalty constraints linked to variational inequalities (VI) is studied with respect to its shape differentiability. The specific problem stemming from quasi-brittle fracture describes an elastic body with a Barenblatt cohesive crack under the inequality condition of non-penetration at the crack faces. Based on the Lagrange approach and using smooth penalization with the Lavrentiev regularization, a formula for the shape derivative is derived. The explicit formula contains both primal and adjoint states and is useful for finding de…
We study necessary and sufficient conditions to attain solutions of set-optimization problems in therms of variational inequalities of Stampacchia and Minty type. The notion of a solution we deal with has been introduced Heyde and Loehne, for convex set-valued objective functions. To define the set-valued variational inequality, we introduce a set-valued directional derivative and we relate it to the Dini derivatives of a family of linearly scalarized problems. The optimality conditions are given by Stampacchia and Minty type Variational inequalities, defined both by the set valued directional…
A method is given for finding roots of a one-variable function using Taylor's expansion of that function and fractional derivative calculated at a suitable tangent point without using Newton's method, but is regarded as a variant of Halley and Newton's one. Several examples regarding polynomials are stated as well. Then, the given method is generalized to functions of several variables belonging to an $n$-dimensional space and one example is given for optimization and solution of a nonlinear system of equations by both our method and Gradient Descent one. A comparison of our method is made wit…
We consider $β$-smooth (satisfies the generalized Holder condition with parameter $β> 2$) stochastic convex optimization problem with zero-order one-point oracle. The best known result was arXiv:2006.07862: $\mathbb{E} \left[f(\overline{x}_N) - f(x^*)\right] = \tilde{\mathcal{O}} \left(\dfrac{n^{2}}{γN^{\frac{β-1}β}} \right)$ in $γ$-strongly convex case, where $n$ is the dimension. In this paper we improve this bound: $\mathbb{E} \left[f(\overline{x}_N) - f(x^*)\right] = \tilde{\mathcal{O}} \left(\dfrac{n^{2-\frac{1}β}}{γN^{\frac{β-1}β}} \right).$
Many engineering problems involve the optimization of computationally expensive models for which derivative information is not readily available. The Bayesian optimization (BO) framework is a particularly promising approach for solving these problems, which uses Gaussian process (GP) models and an expected utility function to systematically tradeoff between exploitation and exploration of the design space. BO, however, is fundamentally limited by the black-box model assumption that does not take into account any underlying problem structure. In this paper, we propose a new algorithm, COBALT, f…
We propose general non-accelerated and accelerated tensor methods under inexact information on the derivatives of the objective, analyze their convergence rate. Further, we provide conditions for the inexactness in each derivative that is sufficient for each algorithm to achieve the desired accuracy. As a corollary, we propose stochastic tensor methods for convex optimization and obtain sufficient mini-batch sizes for each derivative.
A structured version of derivative-free random pattern search optimization algorithms is introduced which is able to exploit coordinate partially separable structure (typically associated with sparsity) often present in unconstrained and bound-constrained optimization problems. This technique improves performance by orders of magnitude and makes it possible to solve large problems that otherwise are totally intractable by other derivative-free methods. A library of interpolation-based modelling tools is also described, which can be associated to the structured or unstructured versions of the i…
We present a level-set based topology optimization algorithm for design optimization problems involving an arbitrary number of different materials, where the evolution of a design is solely guided by topological derivatives. Our method can be seen as an extension of the algorithm that was introduced in (Amstutz, Andrae 2006) for two materials to the case of an arbitrary number $M$ of materials. We represent a design that consists of multiple materials by means of a vector-valued level set function which maps into $\mathbb R^{M-1}$. We divide the space $\mathbb R^{M-1}$ into $M$ sectors, each c…
The topological derivative represents the sensitivity of a domain-dependent functional with respect to a local perturbation of the domain and is a valuable tool in topology optimization. Motivated by an application from electrical engineering, we derive the topological derivative for an optimization problem which is constrained by the quasilinear equation of two-dimensional magnetostatics. Here, the main ingredient is to establish a sufficiently fast decay of the variation of the direct state at scale 1 as $|x|\rightarrow \infty$. In order to apply the method in a bi-directional topology optim…
The paper utilizes Hölder graphical derivatives for characterizing Hölder strong subregularity, isolated calmness and sharp minimum. As applications, we characterize Hölder isolated calmness in linear semi-infinite optimization and Hölder sharp minimizers of some penalty functions for constrained optimization.
We consider the problem of unconstrained minimization of a smooth objective function in $\R^n$ in a setting where only function evaluations are possible. While importance sampling is one of the most popular techniques used by machine learning practitioners to accelerate the convergence of their models when applicable, there is not much existing theory for this acceleration in the derivative-free setting. In this paper, we propose the first derivative free optimization method with importance sampling and derive new improved complexity results on non-convex, convex and strongly convex functions.…
The goal of this paper is to investigate an approach for derivative-free optimization that has not received sufficient attention in the literature and is yet one of the simplest to implement and parallelize. It consists of computing gradients of a smoothed approximation of the objective function (and constraints), and employing them within established codes. These gradient approximations are calculated by finite differences, with a differencing interval determined by the noise level in the functions and a bound on the second or third derivatives. It is assumed that noise level is known or can …
We aim at computing the derivative of the solution to a parametric optimization problem with respect to the involved parameters. For a class broader than that of strongly convex functions, this can be achieved by automatic differentiation of iterative minimization algorithms. If the iterative algorithm converges pointwise, then we prove that the derivative sequence also converges pointwise to the derivative of the minimizer with respect to the parameters. Moreover, we provide convergence rates for both sequences. In particular, we prove that the accelerated convergence rate of the Heavy-ball m…
This paper considers the efficient minimization of the infinite time average of a stationary ergodic process in the space of a handful of design parameters which affect it. Problems of this class, derived from physical or numerical experiments which are sometimes expensive to perform, are ubiquitous in engineering applications. In such problems, any given function evaluation, determined with finite sampling, is associated with a quantifiable amount of uncertainty, which may be reduced via additional sampling. The present paper proposes a new optimization algorithm to adjust the amount of sampl…
Derivative-free optimization methods are numerical methods for optimization problems in which no derivative information is used. Such optimization problems are widely seen in many real applications. One particular class of derivative-free optimization algorithms is trust-region algorithms based on quadratic models given by under-determined interpolation. Different techniques in updating the quadratic model from iteration to iteration will give different interpolation models. In this paper, we propose a new way to update the quadratic model by minimizing the $H^2$ norm of the difference between…
