In this comment, we raise serious concerns over the derivation of the rate of convergence of fractional steepest descent algorithm in Fractional Adaptive Learning (FAL) approach presented in `Fractional Extreme Value Adaptive Training Method: Fractional Steepest Descent Approach' [IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 4, pp. 653--662, April 2015]. We substantiate that the estimate of the rate of convergence is grandiloquent. We also draw attention towards a critical flaw in the design of the algorithm stymieing its applicability for broad adaptive learning problems. Our claims are based on analytical reasoning supported by experimental results.
The aim of this manuscript is to characterize the continuity properties of the multiobjective steepest descent direction for smooth objective functions. We will show that this direction is Holder continuous with optimal exponent 1/2. In particular, this direction fails to be Lipschitz continuous even for polynomial objectives.
This paper investigates some aspects of the variational behaviour of nonsmooth functions, with special emphasis on certain stability phenomena. Relationships linking such properties as sharp minimality, superstability, error bound and sufficiency of first-order optimality conditions are discussed. Their study is performed by employing the steepest descent rate, a rather general tool, which is adequate for a metric space analysis. The positivity of the steepest descent rate is then characterized in terms of $Φ$-subdifferentials. If specialized to a Banach space setting, the resulting characteri…
The limited memory steepest descent method (LMSD) proposed by Fletcher is an extension of the Barzilai-Borwein "two-point step size" strategy for steepest descent methods for solving unconstrained optimization problems. It is known that the Barzilai-Borwein strategy yields a method with an R-linear rate of convergence when it is employed to minimize a strongly convex quadratic. This paper extends this analysis for LMSD, also for strongly convex quadratics. In particular, it is shown that the method is R-linearly convergent for any choice of the history length parameter. The results of numerica…
Generalizing the simplex method, circuit augmentation schemes for linear programs follow circuit directions through the interior of the underlying polyhedron. Steepest-descent augmentation is especially promising, but an implementation of the iterative scheme is a significant challenge. We work towards a viable implementation through a model in which a single linear program is updated dynamically to remain in memory throughout. Computational experiments exhibit dramatic improvements over a naive approach and reveal insight into the next steps required for large-scale computations.
In this paper, we deal with the Front Steepest Descent algorithm for multi-objective optimization. We point out that the algorithm from the literature is often incapable, by design, of spanning large portions of the Pareto front. We thus introduce some modifications within the algorithm aimed to overcome this significant limitation. We prove that the asymptotic convergence properties of the algorithm are preserved and numerically show that the proposed method significantly outperforms the original one.
For solving pseudo-convex global optimization problems, we present a novel fully adaptive steepest descent method (or ASDM) without any hard-to-estimate parameters. For the step-size regulation in an $\varepsilon$-normalized direction, we use the deterministic rules, which were proposed in J. Optim. Theory Appl. (2019,\, DOI: 10.1007/S10957-019-01585-W). We obtained the optimistic convergence estimates for the generated by ASDM sequence of iteration points. Namely, the sequence of function values of iterates has the advantage of the strict monotonic behaviour and globally converges to the obje…
In this paper, we first study nonsmooth steepest descent method for nonsmooth functions defined on Hilbert space and establish the corresponding algorithm by proximal subgradients. Then, we use this algorithm to find stationary points for those functions satisfying prox-regularity and Lipschitzian continuity. As one application, the established algorithm is used to search the minimizer of lower semicontinuous and convex functions on finite-dimensional space. The convergent theorem, as one extension and improvement of the existing converging result for twice continuously differentiable convex f…
The breakthrough ideas in the modern proximal splitting methodologies allow us to express the set of all minimizers of a superposition of multiple nonsmooth convex functions as the fixed point set of computable nonexpansive operators. In this paper, we present practical algorithmic strategies for the hierarchical convex optimization problems which require further strategic selection of a most desirable vector from the solution set of the standard convex optimization. The proposed algorithms are established by applying the hybrid steepest descent method to special nonexpansive operators designe…
Unsupervised clustering of scattered, noisy and high-dimensional data points is an important and difficult problem. Tight continuous relaxations of balanced cut problems have recently been shown to provide excellent clustering results. In this paper, we present an explicit-implicit gradient flow scheme for the relaxed ratio cut problem, and prove that the algorithm converges to a critical point of the energy. We also show the efficiency of the proposed algorithm on the two moons dataset.
This is the first in a series of papers which deal with the development of novel methods for solving a system of linear algebraic equations with a time complexity lower than existing algorithms. The NxN system of linear equations, Ax = b, is often solved iteratively by minimizing the corresponding quadratic form using well known optimization techniques. The simplest of these is the steepest descent method, whose approach to the solution is usually quite rapid at the beginning but slows down drastically after a few iterations. This paper investigates possible approaches which can reduce or avoi…
The steepest descent method for multiobjective optimization on Riemannian manifolds with lower bounded sectional curvature is analyzed in this paper. The aim of the paper is twofold. Firstly, an asymptotic analysis of the method is presented with three different finite procedures for determining the stepsize, namely, Lipschitz stepsize, adaptive stepsize and Armijo-type stepsize. The second aim is to present, by assuming that the Jacobian of the objective function is componentwise Lipschitz continuous, iteration-complexity bounds for the method with these three stepsizes strategies. In additio…
Let $\mathcal{C}$ be the family of compact convex subsets $S$ of the hemisphere in $\rn$ with the property that $S$ contains its dual $S^*;$ let $u\in S^*$, and let $ Φ(S,u)=\frac{2}{ω_n}\int_{S}\ < θ, u \ > \,\, dσ(θ). $ The problem to study $ \inf \big\{Φ(S,u), S \in \mathcal{C}, \, u\in S^* \big\} $ is considered. It is proved that the minima of $ Φ$ are sets of constant width $ π/2 $ with $ u $ on their boundary. More can be said for $n=3$: the minimum set is a Reuleaux triangle on the sphere. The previous problem is related to the one to find the maximal length of steepest descent curves …
Steepest descent is central in variational mathematics. We present a new transparent existence proof for curves of near-maximal slope --- an influential notion of steepest descent in a nonsmooth setting. We moreover show that for semi-algebraic functions --- prototypical nonpathological functions in nonsmooth optimization --- such curves are precisely the solutions of subgradient dynamical systems.
The aim of this paper is to prove that, in an appropriate setting, every iterative sequence generated by the hybrid steepest descent method is convergent whenever so is every iterative sequence generated by the Halpern type iterative method.
The purpose of the paper is to give a complete characterization of the continuity of lower envelopes in the infinite dimensional spaces in terms of the notion of c-regularity. As an application we introduce a variational unconstrained vector optimization problem for smooth functions and characterize when the variational steepest descent directions are continuous in terms of the generating sets which are considered.
Existing implementations of gradient-based optimisation methods typically assume that the problem is posed in Euclidean space. When solving optimality problems on function spaces, the functional derivative is then inaccurately represented with respect to $\ell^2$ instead of the inner product induced by the function space. This error manifests as a mesh dependence in the number of iterations required to solve the optimisation problem. In this paper, an analytic estimate is derived for this iteration count in the case of a simple and generic discretised optimisation problem. The system analysed …
This paper introduces the Fejér-monotone hybrid steepest descent method (FM-HSDM), a new member to the HSDM family of algorithms, for solving affinely constrained minimization tasks in real Hilbert spaces, where convex smooth and non-smooth losses compose the objective function. FM-HSDM offers sequences of estimates which converge weakly and, under certain hypotheses, strongly to solutions of the task at hand. Fixed-point theory, variational inequalities and affine-nonexpansive mappings are utilized to devise a scheme that accommodates affine constraints in a more versatile way than state-of-t…
EPFL（洛桑联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
We propose a new selection rule for the coordinate selection in coordinate descent methods for huge-scale optimization. The efficiency of this novel scheme is provably better than the efficiency of uniformly random selection, and can reach the efficiency of steepest coordinate descent (SCD), enabling an acceleration of a factor of up to $n$, the number of coordinates. In many practical applications, our scheme can be implemented at no extra cost and computational efficiency very close to the faster uniform selection. Numerical experiments with Lasso and Ridge regression show promising improvem…
Steepest descent preconditioning is considered for the recently proposed nonlinear generalized minimal residual (N-GMRES) optimization algorithm for unconstrained nonlinear optimization. Two steepest descent preconditioning variants are proposed. The first employs a line search, while the second employs a predefined small step. A simple global convergence proof is provided for the N-GMRES optimization algorithm with the first steepest descent preconditioner (with line search), under mild standard conditions on the objective function and the line search processes. Steepest descent preconditioni…
In this paper, we study a first order solution method for a particular class of set optimization problems where the solution concept is given by the set approach. We consider the case in which the set-valued objective mapping is identified by a finite number of continuously differentiable selections. The corresponding set optimization problem is then equivalent to find optimistic solutions to vector optimization problems under uncertainty with a finite uncertainty set. We develop optimality conditions for these types of problems, and introduce two concepts of critical points. Furthermore, we p…
This paper introduces the stochastic Fejér-monotone hybrid steepest descent method (S-FM-HSDM) to solve affinely constrained and composite convex minimization tasks. The minimization task is not known exactly; noise contaminates the information about the composite loss function and the affine constraints. S-FM-HSDM generates sequences of random variables that, under certain conditions and with respect to a probability space, converge point-wise to solutions of the noiseless minimization task. S-FM-HSDM enjoys desirable attributes of optimization techniques such as splitting of variables and co…
The paper is concerned with the minimal drag problem in shape optimization of merchant ships exposed to turbulent two-phase flows. Attention is directed to the solution of Reynolds Averaged Navier-Stokes equations using a Finite Volume method. Central aspects are the use of a p-Laplacian relaxed steepest descent direction and the introduction of crucial technical constraints to the optimization procedure, i.e. the center of buoyancy and the displacement of the underwater hull. The example included refers to the frequently investigated Kriso container ship (KCS).
The self regulation problem as an inexact steepest descent method for multicriteria optimization
IME-Universidade Federal de Goiás（戈亚斯联邦大学数学与工程学院）；DM, Universidade Federal do Piauí（皮奥伊联邦大学数学系）；COPPE/Sistemas-Universidade Federal do Rio de Janeiro（里约热内卢联邦大学科佩系统工程研究所）；Aix-Marseille University（艾克斯-马赛大学）
Beijing Jiaotong University（北京交通大学）；The Hong Kong Polytechnic University（香港理工大学）；University of California, Davis（加利福尼亚大学戴维斯分校）
In this paper, we propose a Riemannian smoothing steepest descent method to minimize a nonconvex and non-Lipschitz function on submanifolds. The generalized subdifferentials on Riemannian manifold and the Riemannian gradient sub-consistency are defined and discussed. We prove that any accumulation point of the sequence generated by the Riemannian smoothing steepest descent method is a stationary point associated with the smoothing function employed in the method, which is necessary for the local optimality of the original non-Lipschitz problem. Under the Riemannian gradient sub-consistency con…
The steepest descent method proposed by Fliege et al. motivates the research on descent methods for multiobjective optimization, which has received increasing attention in recent years. However, empirical results show that the Armijo line search often gives a very small stepsize along the steepest direction, which decelerates the convergence seriously. This paper points out that the issue is mainly due to the imbalances among objective functions. To address this issue, we propose a Barzilai-Borwein descent method for multiobjective optimization (BBDMO) that dynamically tunes gradient magnitude…
In this work, we discuss the task of finding a direction of optimal descent for problems in Shape Optimisation and its relation to the dual problem in Optimal Transport. This link was first observed in a previous work which sought minimisers of a shape derivative over the space of Lipschitz functions which may be closely related to the $\infty$-Laplacian. We provide some results of shape optimisation using this novel Lipschitz approach, highlighting the difference between the Lipschitz and $W^{1,\infty}$ semi-norms. After this, we provide an overview of the necessary results from Optimal trans…
Refining asymptotic complexity bounds for nonconvex optimization methods, including why steepest descent is $o(ε^{-2})$ rather than $\mathcal{O}(ε^{-2})$
Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)（人工智能基础研究实验室（BRAIn Lab））；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（人工智能 Mohamed bin Zayed 大学）；Innopolis University（Innopolis 大学）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）；EPFL（洛桑联邦理工学院）
Two popular examples of first-order optimization methods over linear spaces are coordinate descent and matching pursuit algorithms, with their randomized variants. While the former targets the optimization by moving along coordinates, the latter considers a generalized notion of directions. Exploiting the connection between the two algorithms, we present a unified analysis of both, providing affine invariant sublinear $\mathcal{O}(1/t)$ rates on smooth objectives and linear convergence on strongly convex objectives. As a byproduct of our affine invariant analysis of matching pursuit, our rates…
