This paper introduces a novel algorithmic framework for a deep neural network (DNN), which in a mathematically rigorous manner, allows us to incorporate history (or memory) into the network -- it ensures all layers are connected to one another. This DNN, called Fractional-DNN, can be viewed as a time-discretization of a fractional in time nonlinear ordinary differential equation (ODE). The learning problem then is a minimization problem subject to that fractional ODE as constraints. We emphasize that an analogy between the existing DNN and ODEs, with standard time derivative, is well-known by now. The focus of our work is the Fractional-DNN. Using the Lagrangian approach, we provide a derivation of the backward propagation and the design equations. We test our network on several datasets for classification problems. Fractional-DNN offers various advantages over the existing DNN. The key benefits are a significant improvement to the vanishing gradient issue due to the memory effect, and better handling of nonsmooth data due to the network's ability to approximate non-smooth functions.
Recent studies on diffusion-based sampling methods have shown that Langevin Monte Carlo (LMC) algorithms can be beneficial for non-convex optimization, and rigorous theoretical guarantees have been proven for both asymptotic and finite-time regimes. Algorithmically, LMC-based algorithms resemble the well-known gradient descent (GD) algorithm, where the GD recursion is perturbed by an additive Gaussian noise whose variance has a particular form. Fractional Langevin Monte Carlo (FLMC) is a recently proposed extension of LMC, where the Gaussian noise is replaced by a heavy-tailed α-stable noise. …
We propose a novel Caputo fractional derivative-based optimization algorithm. Upon defining the Caputo fractional gradient with respect to the Cartesian coordinate, we present a generic Caputo fractional gradient descent (CFGD) method. We prove that the CFGD yields the steepest descent direction of a locally smoothed objective function. The generic CFGD requires three parameters to be specified, and a choice of the parameters yields a version of CFGD. We propose three versions -- non-adaptive, adaptive terminal and adaptive order. By focusing on quadratic objective functions, we provide a conv…
Time-varying non-convex continuous-valued non-linear constrained optimization is a fundamental problem. We study conditions wherein a momentum-like regularising term allow for the tracking of local optima by considering an ordinary differential equation (ODE). We then derive an efficient algorithm based on a predictor-corrector method, to track the ODE solution.
Distributed Fractional Bayesian Learning for Adaptive Optimization
Department of Control Science and Engineering, Tongji University（控制科学与工程系，同济大学）；State Key Laboratory of Autonomous Intelligent Unmanned Systems（自主智能无人系统国家重点实验室）；Frontiers Science Center for Intelligent Autonomous Systems, Ministry of Education（智能自主系统前沿科学中心，教育部）；Shanghai Institute of Intelligent Science and Technology, Tongji University（上海智能科学与技术研究院，同济大学）
SINTEF（挪威科学与工业研究基金会）；ETH Zürich（苏黎世联邦理工学院）；University of Southern California（南加州大学）；TIFR Centre for Applicable Mathematics（塔塔基础研究院应用数学中心）
We present a novel active learning algorithm, termed as iterative surrogate model optimization (ISMO), for robust and efficient numerical approximation of PDE constrained optimization problems. This algorithm is based on deep neural networks and its key feature is the iterative selection of training data through a feedback loop between deep neural networks and any underlying standard optimization algorithm. Under suitable hypotheses, we show that the resulting optimizers converge exponentially fast (and with exponentially decaying variance), with respect to increasing number of training sample…
Syracuse University（雪城大学）；Georgia Institute of Technology（佐治亚理工学院）；Harvard University（哈佛大学）
This paper surveys the recent attempts at leveraging machine learning to solve constrained optimization problems. It focuses on surveying the work on integrating combinatorial solvers and optimization methods with machine learning architectures. These approaches hold the promise to develop new hybrid machine learning and optimization methods to predict fast, approximate, solutions to combinatorial problems and to enable structural logical inference. This paper presents a conceptual review of the recent advancements in this emerging area.
In this paper, we introduce an Augmented Lagrangian based method to incorporate the multiple objectives (MO) in a search ranking algorithm. Optimizing MOs is an essential and realistic requirement for building ranking models in production. The proposed method formulates MO in constrained optimization and solves the problem in the popular Boosting framework -- a novel contribution of our work. Furthermore, we propose a procedure to set up all optimization parameters in the problem. The experimental results show that the method successfully achieves MO criteria much more efficiently than existin…
Many clustering applications in machine learning and data mining rely on solving metric-constrained optimization problems. These problems are characterized by $O(n^3)$ constraints that enforce triangle inequalities on distance variables associated with $n$ objects in a large dataset. Despite its usefulness, metric-constrained optimization is challenging in practice due to the cubic number of constraints and the high-memory requirements of standard optimization software. Recent work has shown that iterative projection methods are able to solve metric-constrained optimization problems on a much …
This paper presents a constrained-optimization formulation for the prioritized execution of learned robot tasks. The framework lends itself to the execution of tasks encoded by value functions, such as tasks learned using the reinforcement learning paradigm. The tasks are encoded as constraints of a convex optimization program by using control Lyapunov functions. Moreover, an additional constraint is enforced in order to specify relative priorities between the tasks. The proposed approach is showcased in simulation using a team of mobile robots executing coordinated multi-robot tasks.
In this paper, we employ a reduced basis method for solving the PDE constrained optimization problem governed by a fractional parabolic equation with the fractional derivative in time from order beta in (0,1) is defined by Caputo fractional derivative.
Saarland University（萨尔大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
The (constrained) minimization of a ratio of set functions is a problem frequently occurring in clustering and community detection. As these optimization problems are typically NP-hard, one uses convex or spectral relaxations in practice. While these relaxations can be solved globally optimally, they are often too loose and thus lead to results far away from the optimum. In this paper we show that every constrained minimization problem of a ratio of non-negative set functions allows a tight relaxation into an unconstrained continuous optimization problem. This result leads to a flexible framew…
Isoperimetric problems consist in minimizing or maximizing a cost functional subject to an integral constraint. In this work, we present two fractional isoperimetric problems where the Lagrangian depends on a combined Caputo derivative of variable fractional order and we present a new variational problem subject to a holonomic constraint. We establish necessary optimality conditions in order to determine the minimizers of the fractional problems. The terminal point in the cost integral, as well the terminal state, are considered to be free, and we obtain corresponding natural boundary conditio…
The existence and stability for a class of fuzzy fractional boundary value problems constrained by variational inequalities
Luoyang Normal University（洛阳师范学院）；Southwest Petroleum University（西南石油大学）；University of Electronic Science and Technology of China（电子科技大学）；Sichuan University（四川大学）
In this note we present a multigrid preconditioning method for solving quadratic optimization problems constrained by a fractional diffusion equation. Multigrid methods within the all-at-once approach to solve the first order-order optimality Karush-Kuhn-Tucker (KKT) systems are widely popular, but their development have relied on the underlying systems being sparse. On the other hand, for most discretizations, the matrix representation of fractional operators is expected to be dense. We develop a preconditioning strategy for our problem based on a reduced approach, namely we eliminate the sta…
Based on a method introduced by Leitmann [Internat. J. Non-Linear Mech. {\bf 2} (1967), 55--59], we exhibit exact solutions for some fractional optimization problems of the calculus of variations and optimal control.
Several constrained optimization problems have been adequately solved over the years thanks to advances in the metaheuristics area. In this paper, we evaluate a novel self-adaptive and auto-constructive metaheuristic called Drone Squadron Optimization (DSO) in solving constrained engineering design problems. This paper evaluates DSO with death penalty on three widely tested engineering design problems. Results show that the proposed approach is competitive with some very popular metaheuristics.
George Mason University（乔治梅森大学）；Argonne National Laboratory（阿贡国家实验室）
In this work we consider a generalized bilevel optimization framework for solving inverse problems. We introduce fractional Laplacian as a regularizer to improve the reconstruction quality, and compare it with the total variation regularization. We emphasize that the key advantage of using fractional Laplacian as a regularizer is that it leads to a linear operator, as opposed to the total variation regularization which results in a nonlinear degenerate operator. Inspired by residual neural networks, to learn the optimal strength of regularization and the exponent of fractional Laplacian, we de…
This paper deals with Pareto solutions of a nonsmooth fractional interval-valued multiobjective optimization. We first introduce four types of Pareto solutions of the considered problem by considering the lower-upper interval order relation and then apply some advanced tools of variational analysis and generalized differentiation to establish necessary optimality conditions for these solutions. Sufficient conditions for Pareto solutions of such a problem are also provided by means of introducing the concepts of (strictly) generalized convex functions defined in terms of the limiting/Mordukhovi…
A method is given for finding roots of a one-variable function using Taylor's expansion of that function and fractional derivative calculated at a suitable tangent point without using Newton's method, but is regarded as a variant of Halley and Newton's one. Several examples regarding polynomials are stated as well. Then, the given method is generalized to functions of several variables belonging to an $n$-dimensional space and one example is given for optimization and solution of a nonlinear system of equations by both our method and Gradient Descent one. A comparison of our method is made wit…
Nonlinear constrained optimization problems are encountered in many scientific fields. To utilize the huge calculation power of current computers, many mathematic models are also rebuilt as optimization problems. Most of them have constrained conditions which need to be handled. Borrowing biological concepts, a study is accomplished for dealing with the constraints in the synthesis of a four-bar mechanism. Biologically regarding the constrained condition as a form of selection for characteristics of a population, four new algorithms are proposed, and a new explanation is given for the penalty …
We extend Robust Optimization to fractional programming, where both the objective and the constraints contain uncertain parameters. Earlier work did not consider uncertainty in both the objective and the constraints, or did not use Robust Optimization. Our contribution is threefold. First, we provide conditions to guarantee that either a globally optimal solution, or a sequence converging to the globally optimal solution, can be found by solving one or more convex optimization problems. Second, we identify two cases for which an exact solution can be obtained by solving a single optimization p…
Nonlocal operators of fractional type are a popular modeling choice for applications that do not adhere to classical diffusive behavior; however, one major challenge in nonlocal simulations is the selection of model parameters. In this work we propose an optimization-based approach to parameter identification for fractional models with an optional truncation radius. We formulate the inference problem as an optimal control problem where the objective is to minimize the discrepancy between observed data and an approximate solution of the model, and the control variables are the fractional order …
This paper presents reproducing kernel Hilbert spaces method to obtain the numerical solution for partial differential equation constrained optimization problem.
Motivated by gradient methods in optimization theory, we give methods based on $ψ$-fractional derivatives of order $α$ in order to solve unconstrained optimization problems. The convergence of these methods is analyzed in detail. This paper also presents an Adams-Bashforth-Moulton (ABM) method for the estimation of solutions to equations involving $ψ$-fractional derivatives. Numerical examples using the ABM method show that the fractional order $α$ and weight $ψ$ are tunable parameters, which can be helpful for improving the performance of gradient descent methods.
One of the most important optimality conditions to aid to solve a vector optimization problem is the first-order necessary optimality condition that generalizes the Karush-Kuhn-Tucker condition. However, to obtain the sufficient optimality conditions, it is necessary to impose additional assumptions on the objective functions and in the constraint set. The present work is concerned with the constrained vector quadratic fractional optimization problem. It shows that sufficient Pareto optimality conditions and the main duality theorems can be established without the assumption of generalized con…
This paper presents the main characteristics of the evolutionary optimization code named EOS, Evolutionary Optimization at Sapienza, and its successful application to challenging, real-world space trajectory optimization problems. EOS is a global optimization algorithm for constrained and unconstrained problems of real-valued variables. It implements a number of improvements to the well-known Differential Evolution (DE) algorithm, namely, a self-adaptation of the control parameters, an epidemic mechanism, a clustering technique, an $\varepsilon$-constrained method to deal with nonlinear constr…
In this note we study multiple-ratio fractional 0--1 programs, a broad class of NP-hard combinatorial optimization problems. In particular, under some relatively mild assumptions we provide a complete characterization of the conditions, which ensure that a single-ratio function is submodular. Then we illustrate our theoretical results with the assortment optimization and facility location problems, and discuss practical situations that guarantee submodularity in the considered application settings. In such cases, near-optimal solutions for multiple-ratio fractional 0--1 programs can be found v…
We propose an implicit iterative algorithm for an exact penalty method arising from inequality constrained optimization problems. A rapidly convergent fixed point method is developed for a regularized penalty functional. The applicability and feasibility of the proposed method is demonstrated using large scale inequality constrained problems.
The aim of this study is to find the optimum of a linear fractional function over the efficient set of a multi-objective linear fractional integer program without generating all efficient solutions. By its nature, it is a global optimization problem since the efficient set is discrete, hence not convex. For this purpose, a branch and bound based method is described with a double mission to search for an optimal solution for a given linear fractional function which is moreover, efficient for a multi-objective linear fractional integer programming problem. Tests performed on instances randomly g…
We introduce an extended tangent cone of high order to a set and study its properties. Then we use this local approximation for deriving high-order necessary conditions for local minimizers of constrained optimization problems.
Institute of Artificial Intelligence, MIT World Peace University（MIT世界和平大学人工智能研究所）；Artificial Intelligence Research Institute, University of Veracruz（韦拉克鲁斯大学人工智能研究所）
Several Artificial Intelligence based heuristic and metaheuristic algorithms have been developed so far. These algorithms have shown their superiority towards solving complex problems from different domains. However, it is necessary to critically validate these algorithms for solving real-world constrained optimization problems. The search behavior in those problems is different as it involves large number of linear, nonlinear and non-convex type equality and inequality constraints. In this work a 57 real-world constrained optimization problems test suite is solved using two constrained metahe…
In Hilbert space, we propose a family of primal-dual dynamical system for affine constrained convex optimization problem. Several damping coefficients, time scaling coefficients, and perturbation terms are thus considered. By constructing the energy functions, we investigate the convergence rates with different choices of the damping coefficients and time scaling coefficients. Our results extend the inertial dynamical approaches for unconstrained convex optimization problems to affine constrained convex optimization problems.
DY Gao together with some of his collaborators applied his Canonical duality theory (CDT) for solving a class of constrained optimization problems. Unfortunately, in several papers on this subject there are unclear statements, not convincing proofs, or even false results. It is our aim in this work to study rigorously these class of constrained optimization problems in finite dimensional spaces and to discuss several results published in the last ten years.
In this paper, we prove that the global version of the $Ł$ojasiewicz gradient inequality holds for quadratic sphere constrained optimization problem with exponent $θ=\frac{3}{4}$. An example from Ting Kei Pong shows that $θ=\frac{3}{4}$ is tight. This is the first $Ł$ojasiewicz gradient inequality established for the sphere constrained optimization problem with a linear term.
Motivated by TRACE algorithm [Curtis et al. 2017], we propose a trust region algorithm for finding second order stationary points of a linearly constrained non-convex optimization problem. We show the convergence of the proposed algorithm to (ε_g, ε_H)-second order stationary points in \widetilde{\mathcal{O}}(\max{ε_g^{-3/2}, ε_H^{-3}}) iterations. This iteration complexity is achieved for general linearly constrained optimization without cubic regularization of the objective function.
We consider the constrained assortment optimization problem under the mixed multinomial logit model. Even moderately sized instances of this problem are challenging to solve directly using standard mixed-integer linear optimization formulations. This has motivated recent research exploring customized optimization strategies and approximation techniques. In contrast, we develop a novel conic quadratic mixed-integer formulation. This new formulation, together with McCormick inequalities exploiting the capacity constraints, enables the solution of large instances using commercial optimization sof…
