We present a generic solver for unconstrained control problems (UCPs) whose objectives take the form of an integral functional of the controllers. The solver generalizes and improves upon the algorithm proposed by Tseng and Tang for the Witsenhausen's counterexample, which provides the best-known results. In essence, we show that minimizing the objective implies minimizing the marginal cost functions almost everywhere, and we perform the latter task pointwisely by the adaptive minimization technique, which speeds up the computation. We implement single-threaded and parallelized versions of the proposed algorithm. Our implementation runs $30 \times$ faster than Tseng and Tang's algorithm on the Witsenhausen's counterexample, and we demonstrate the applicability of the solver and discuss the possible generalization to constrained problems through two more examples.
Recently, inspired by quantum annealing, many solvers specialized for unconstrained binary quadratic programming problems have been developed. For further improvement and application of these solvers, it is important to clarify the differences in their performance for various types of problems. In this study, the performance of four quadratic unconstrained binary optimization problem solvers, namely D-Wave Hybrid Solver Service (HSS), Toshiba Simulated Bifurcation Machine (SBM), Fujitsu DigitalAnnealer (DA), and simulated annealing on a personal computer, was benchmarked. The problems used for…
An adaptive regularization algorithm for unconstrained nonconvex optimization is proposed that is capable of handling inexact objective-function and derivative values, and also of providing approximate minimizer of arbitrary order. In comparison with a similar algorithm proposed in Cartis, Gould, Toint (2021), its distinguishing feature is that it is based on controlling the relative error between the model and objective values. A sharp evaluation complexity complexity bound is derived for the new algorithm.
In this paper, we focus on the nonconvex-strongly-convex bilevel optimization problem (BLO). In this BLO, the objective function of the upper-level problem is nonconvex and possibly nonsmooth, and the lower-level problem is smooth and strongly convex with respect to the underlying variable $y$. We show that the feasible region of BLO is a Riemannian manifold. Then we transform BLO to its corresponding unconstrained constraint dissolving problem (CDB), whose objective function is explicitly formulated from the objective functions in BLO. We prove that BLO is equivalent to the unconstrained opti…
We investigate the unconstrained global optimization of functions with low effective dimensionality, that are constant along certain (unknown) linear subspaces. Extending the technique of random subspace embeddings in [Wang et al., Bayesian optimization in a billion dimensions via random embeddings. JAIR, 55(1): 361--387, 2016], we study a generic Random Embeddings for Global Optimization (REGO) framework that is compatible with any global minimization algorithm. Instead of the original, potentially large-scale optimization problem, within REGO, a Gaussian random, low-dimensional problem with …
We consider minimization of a smooth nonconvex objective function using an iterative algorithm based on Newton's method and the linear conjugate gradient algorithm, with explicit detection and use of negative curvature directions for the Hessian of the objective function. The algorithm tracks Newton-conjugate gradient procedures developed in the 1980s closely, but includes enhancements that allow worst-case complexity results to be proved for convergence to points that satisfy approximate first-order and second-order optimality conditions. The complexity results match the best known results in…
In this article we investigate the possibilities of accelerating the double smoothing technique when solving unconstrained nondifferentiable convex optimization problems. This approach relies on the regularization in two steps of the Fenchel dual problem associated to the problem to be solved into an optimization problem having a differentiable strongly convex objective function with Lipschitz continuous gradient. The doubly regularized dual problem is then solved via a fast gradient method. The aim of this paper is to show how do the properties of the functions in the objective of the primal …
Annealed Mean Field Descent Is Highly Effective for Quadratic Unconstrained Binary Optimization
Institute of Integrated Research, Institute of Science Tokyo（东京科学大学综合研究学院）；School of Engineering, Institute of Science Tokyo（东京科学大学工程学院）；Faculty of Science and Engineering, Waseda University（早稻田大学理工学部）
Finding the minimum of a multivariate real polynomial is a well-known hard problem with various applications. We present a polynomial time algorithm to approximate such lower bounds via sums of nonnegative circuit polynomials (SONC). As a main result, we carry out the first large-scale comparison of SONC, using this algorithm and different geometric programming (GP) solvers, with the classical sums of squares (SOS) approach, using several of the most common semidefinite programming (SDP) solvers. SONC yields bounds competitive to SOS in several cases, but using significantly less time and memo…
In this paper (part 1), we describe a derivative-free trust-region method for solving unconstrained optimization problems. We will discuss a method when we relax the model order assumption and use artificial neural network techniques to build a computationally relatively inexpensive model. We directly find an estimate of the objective function minimizer without explicitly constructing a model function. Therefore, we need to have the neural-network model derivatives, which can be obtained simply through a back-propagation process.
In this paper, we propose a hybrid framework to solve large-scale permutation-based combinatorial problems effectively using a high-performance quadratic unconstrained binary optimization (QUBO) solver. To do so, transformations are required to change a constrained optimization model to an unconstrained model that involves parameter tuning. We propose techniques to overcome the challenges in using a QUBO solver that typically comes with limited numbers of bits. First, to smooth the energy landscape, we reduce the magnitudes of the input without compromising optimality. We propose a machine lea…
A class of non-convex optimization problems with DC objective function is studied, where DC stands for being representable as the difference $f=g-h$ of two convex functions $g$ and $h$. In particular, we deal with the special case where one of the two convex functions $g$ or $h$ is polyhedral. In case $g$ is polyhedral, we show that a solution of the DC program can be obtained from a solution of an associated polyhedral projection problem. In case $h$ is polyhedral, we prove that a solution of the DC program can be obtained by solving a polyhedral projection problem and finitely many convex pr…
In this paper, we propose a variable metric method for unconstrained multiobjective optimization problems (MOPs). First, a sequence of points is generated using different positive definite matrices in the generic framework. It is proved that accumulation points of the sequence are Pareto critical points. Then, without convexity assumption, strong convergence is established for the proposed method. Moreover, we use a common matrix to approximate the Hessian matrices of all objective functions, along which, a new nonmonotone line search technique is proposed to achieve a local superlinear conver…
We propose a new globalization strategy that can be used in unconstrained optimization algorithms to support rapid convergence from remote starting points. Our approach is based on using multiple points at each iteration to build a representative model of the objective function. Using the new information gathered from those multiple points, a local step is gradually improved by updating its direction as well as its length. We give a global convergence result and also provide parallel implementation details accompanied with a numerical study. Our numerical study shows that the proposed algorith…
Digital Annealer (DA) is a computer architecture designed for tackling combinatorial optimization problems formulated as quadratic unconstrained binary optimization (QUBO) models. In this paper, we present the results of an extensive computational study to evaluate the performance of DA in a systematic way in comparison to multiple state-of-the-art solvers for different problem classes. We examine pure QUBO models, as well as QUBO reformulations of three constrained problems, namely quadratic assignment, quadratic cycle partition, and selective graph coloring, with the last two being new appli…
The article deals with operations defined on convex polyhedra or polyhedral convex functions. Given two convex polyhedra, operations like Minkowski sum, intersection and closed convex hull of the union are considered. Basic operations for one convex polyhedron are, for example, the polar, the conical hull and the image under affine transformation. The concept of a P-representation of a convex polyhedron is introduced. It is shown that many polyhedral calculus operations can be expressed explicitly in terms of P-representations. We point out that all the relevant computational effort for polyhe…
Mohammed VI Polytechnic University（穆罕默德六世理工大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
In this paper, we propose a new zero order optimization method called minibatch stochastic three points (MiSTP) method to solve an unconstrained minimization problem in a setting where only an approximation of the objective function evaluation is possible. It is based on the recently proposed stochastic three points (STP) method (Bergou et al., 2020). At each iteration, MiSTP generates a random search direction in a similar manner to STP, but chooses the next iterate based solely on the approximation of the objective function rather than its exact evaluations. We also analyze our method's comp…
In this brief, we consider the constrained optimization problem underpinning model predictive control (MPC). We show that this problem can be decomposed into an unconstrained optimization problem with the same cost function as the original problem and a constrained optimization problem with a modified cost function and dynamics that have been precompensated according to the solution of the unconstrained problem. In the case of linear systems subject to a quadratic cost, the unconstrained problem has the familiar LQR solution and the constrained problem reduces to a minimum-norm projection. Thi…
Quadratic Unconstrained Binary Optimization models are useful for solving a diverse range of optimization problems. Constraints can be added by incorporating quadratic penalty terms into the objective, often with the introduction of slack variables needed for conversion of inequalities. This transformation can lead to a significant increase in the size and density of the problem. Herein, we propose an efficient approach for recasting inequality constraints that reduces the number of linear and quadratic variables. Experimental results illustrate the efficacy.
Higher-Order vs. Quadratic Binary Optimization: Which Is Better for Probability Optimization with Tensor Sampling?
Research Center of the Artificial Intelligence Institute, Innopolis University（因诺波利斯大学人工智能研究所研究中心）；Innopolis University（因诺波利斯大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Q Deep
We propose a method for low-rank semidefinite programming in application to the semidefinite relaxation of unconstrained binary quadratic problems. The method improves an existing solution of the semidefinite programming relaxation to achieve a lower rank solution. This procedure is computationally efficient as it does not require projecting on the cone of positive-semidefinite matrices. Its performance in terms of objective improvement and rank reduction is tested over multiple graphs of large-scale Gset graph collection and over binary optimization problems from the Biq Mac collection.
The routing and wavelength assignment with protection is an important problem in telecommunications. Given an optical network and incoming connection requests, a commonly studied variant of the problem aims to grant maximum number of requests by assigning lightpaths at minimum network resource usage level, while ensuring the provided services remain functional in case of a single-link failure through dedicated path protection. We consider a practically relevant version where alternative lightpaths for requests are assumed to be given as a precomputed set, and show that it is NP-hard. We formul…
We consider a continuous-time optimization method based on a dynamical system, where a massive particle starting at rest moves in the conservative force field generated by the objective function, without any kind of friction. We formulate a restart criterion based on the mean dissipation of the kinetic energy, and we prove a global convergence result for strongly-convex functions. Using the Symplectic Euler discretization scheme, we obtain an iterative optimization algorithm. We have considered a discrete mean dissipation restart scheme, but we have also introduced a new restart procedure base…
We present DFO-LS, a software package for derivative-free optimization (DFO) for nonlinear Least-Squares (LS) problems, with optional bound constraints. Inspired by the Gauss-Newton method, DFO-LS constructs simplified linear regression models for the residuals. DFO-LS allows flexible initialization for expensive problems, whereby it can begin making progress from as few as two objective evaluations. Numerical results show DFO-LS can gain reasonable progress on some medium-scale problems with fewer objective evaluations than is needed for one gradient evaluation. DFO-LS has improved robustness…
Kyushu Sangyo University（九州产业大学）；KLab Inc.（KLab公司）；RIKEN（理化学研究所）；Tokyo Institute of Technology（东京工业大学）
A multi-objective optimization problem is $C^r$ weakly simplicial if there exists a $C^r$ surjection from a simplex onto the Pareto set/front such that the image of each subsimplex is the Pareto set/front of a subproblem, where $0\leq r\leq \infty$. This property is helpful to compute a parametric-surface approximation of the entire Pareto set and Pareto front. It is known that all unconstrained strongly convex $C^r$ problems are $C^{r-1}$ weakly simplicial for $1\leq r \leq \infty$. In this paper, we show that all unconstrained strongly convex problems are $C^0$ weakly simplicial. The usefuln…
The unconstrained minimization of a sufficiently smooth objective function $f(x)$ is considered, for which derivatives up to order $p$, $p\geq 2$, are assumed to be available. An adaptive regularization algorithm is proposed that uses Taylor models of the objective of order $p$ and that is guaranteed to find a first- and second-order critical point in at most $O \left(\max\left( ε_1^{-\frac{p+1}{p}}, ε_2^{-\frac{p+1}{p-1}} \right) \right)$ function and derivatives evaluations, where $ε_1$ and $ε_2 >0$ are prescribed first- and second-order optimality tolerances. Our approach extends the method…
This paper focus on the minimization of a possibly nonsmooth objective function over the Stiefel manifold. The existing approaches either lack efficiency or can only tackle prox-friendly objective functions. We propose a constraint dissolving function named NCDF and show that it has the same first-order stationary points and local minimizers as the original problem in a neighborhood of the Stiefel manifold. Furthermore, we show that the Clarke subdifferential of NCDF is easy to achieve from the Clarke subdifferential of the objective function. Therefore, various existing approaches for unconst…
PETScML: Second-order solvers for training regression problems in Scientific Machine Learning
Extreme Computing Research Center, King Abdullah University of Science and Technology（阿卜杜拉国王科技大学极端计算研究中心）；Mathematical Institute, University of Oxford（牛津大学数学研究所）
In this paper, we propose a new and efficient nonmonotone adaptive trust region algorithm to solve unconstrained optimization problems. This algorithm incorporates two novelties: it benefits from a radius dependent shrinkage parameter for adjusting the trust region radius that avoids undesirable directions and it exploits a new strategy to prevent sudden increments of objective function values in nonmonotone trust region techniques. Global convergence of this algorithm is investigated under some mild conditions. Numerical experiments demonstrate the efficiency and robustness of the proposed al…
A q-Gauss-Newton algorithm is an iterative procedure that solves nonlinear unconstrained optimization problems based on minimization of the sum squared errors of the objective function residuals. Main advantage of the algorithm is that it approximates matrix of q-second order derivatives with the first-order q-Jacobian matrix. For that reason, the algorithm is much faster than q-steepest descent algorithms. The convergence of q-GN method is assured only when the initial guess is close enough to the solution. In this paper the influence of the parameter q to the non-linear problem solving is pr…
Leveraging Classical and Quantum Computing for Process Systems Engineering Applications: Decomposition Algorithm with Ising Solvers for Efficient Discrete Landscape Exploration
Davidson School of Chemical Engineering, Purdue University（普渡大学戴维森化学工程学院）
Proper efficiency results in vector optimisation in real linear-topological spaces based on vectorial penalisation
Leibniz Universität Hannover（汉诺威莱布尼茨大学）；Norwegian University of Science and Technology（挪威科技大学）；Martin-Luther-University Halle-Wittenberg（马丁·路德·哈勒-维滕贝格大学）
In this article we show that the boundary of the Pareto critical set of an unconstrained multiobjective optimization problem (MOP) consists of Pareto critical points of subproblems considering subsets of the objective functions. If the Pareto critical set is completely described by its boundary (e.g. if we have more objective functions than dimensions in the parameter space), this can be used to solve the MOP by solving a number of MOPs with fewer objective functions. If this is not the case, the results can still give insight into the structure of the Pareto critical set. This technique is es…
Institute of High Performance Computing, Agency for Science Technology and Research（新加坡科技研究局高性能计算研究所）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）
An increasingly popular method for solving a constrained combinatorial optimisation problem is to first convert it into a quadratic unconstrained binary optimisation (QUBO) problem, and solve it using a standard QUBO solver. However, this relaxation introduces hyper-parameters that balance the objective and penalty terms for the constraints, and their chosen values significantly impact performance. Hence, tuning these parameters is an important problem. Existing generic hyper-parameter tuning methods require multiple expensive calls to a QUBO solver, making them impractical for performance cri…
We propose a framework to use Nesterov's accelerated method for constrained convex optimization problems. Our approach consists of first reformulating the original problem as an unconstrained optimization problem using a continuously differentiable exact penalty function. This reformulation is based on replacing the Lagrange multipliers in the augmented Lagrangian of the original problem by Lagrange multiplier functions. The expressions of these Lagrange multiplier functions, which depend upon the gradients of the objective function and the constraints, can make the unconstrained penalty funct…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
This manuscript proposes a probabilistic framework for algorithms that iteratively solve unconstrained linear problems $Bx = b$ with positive definite $B$ for $x$. The goal is to replace the point estimates returned by existing methods with a Gaussian posterior belief over the elements of the inverse of $B$, which can be used to estimate errors. Recent probabilistic interpretations of the secant family of quasi-Newton optimization algorithms are extended. Combined with properties of the conjugate gradient algorithm, this leads to uncertainty-calibrated methods with very limited cost overhead o…
