We present a regularization method to approach a solution of the pessimistic formulation of ill -posed bilevel problems . This allows to overcome the difficulty arising from the non uniqueness of the lower level problems solutions and responses. We prove existence of approximated solutions, give convergence result using Hoffman-like assumptions. We end with objective value error estimates.
This paper is concerned with an optimization problem governed by the Kantorovich optimal transportation problem. This gives rise to a bilevel optimization problem, which can be reformulated as a mathematical problem with complementarity constraints in the space of regular Borel measures. Because of the non-smoothness induced by the complementarity relations, problems of this type are frequently regularized. Here we apply a quadratic regularization of the Kantorovich problem. As the title indicates, this is the first part in a series of three papers. It addresses the existence of optimal soluti…
This paper is concerned with an optimization problem that is constrained by the Kantorovich optimal transportation problem. This bilevel optimization problem can be reformulated as a mathematical problem with complementarity constraints in the space of regular Borel measures. Because of the non-smoothness induced by the complementarity relations, problems of this type are frequently regularized. Here we apply a quadratic regularization of the Kantorovich problem. As the title indicates, this is the second part in a series of three papers. While the existence of optimal solutions to both the bi…
We address the problem of optimal scale-dependent parameter learning in total variation image denoising. Such problems are formulated as bilevel optimization instances with total variation denoising problems as lower-level constraints. For the bilevel problem, we are able to derive M-stationarity conditions, after characterizing the corresponding Mordukhovich generalized normal cone and verifying suitable constraint qualification conditions. We also derive B-stationarity conditions, after investigating the Lipschitz continuity and directional differentiability of the lower-level solution opera…
Total Generalized Variation (TGV) regularization in image reconstruction relies on an infimal convolution type combination of generalized first- and second-order derivatives. This helps to avoid the staircasing effect of Total Variation (TV) regularization, while still preserving sharp contrasts in images. The associated regularization effect crucially hinges on two parameters whose proper adjustment represents a challenging task. In this work, a bilevel optimization framework with a suitable statistics-based upper level objective is proposed in order to automatically select these parameters. …
Whiteness-based bilevel learning of regularization parameters in imaging
University of Insubria（因苏布里亚大学）；Laboratoire I3S（I3S实验室）；CNRS（法国国家科学研究中心）；UniCA（尼斯蔚蓝海岸大学）；Inria（法国国家信息与自动化研究所）；University of Naples Federico II（那不勒斯费德里科二世大学）；University of Bologna（博洛尼亚大学）
In this paper, we exploit the so-called value function reformulation of the bilevel optimization problem to develop duality results for the problem. Our approach builds on Fenchel-Lagrange-type duality to establish suitable results for the bilevel optimization problem. First, we overview some standard duality results to show that they are not applicable to our problem. Secondly, via the concept of partial calmness, we establish weak and strong duality results. In particular, Lagrange, Fenchel-Lagrange, and Toland-Fenchel- Lagrange duality concepts are investigated for this type of problems und…
Motivated by emerging applications in wireless sensor networks and large-scale data processing, we consider distributed optimization over directed networks where the agents communicate their information locally to their neighbors to cooperatively minimize a global cost function. We introduce a new unifying distributed constrained optimization model that is characterized as a bilevel optimization problem. This model captures a wide range of existing problems over directed networks including: (i) Distributed optimization with linear constraints; (ii) Distributed unconstrained nonstrongly convex …
A Single-Loop Gradient Algorithm for Pessimistic Bilevel Optimization via Smooth Approximation
Southern University of Science and Technology（南方科技大学）；National Center for Applied Mathematics Shenzhen, Southern University of Science and Technology（深圳应用数学国家中心）；Detection Institute for Advanced Technology Longhua-Shenzhen (DIATLHSZ)（龙华-深圳先进技术检测研究院）
University of Bath（巴斯大学）；CNRS（法国国家科学研究中心）；Ecole Normale Supérieure, PSL University（巴黎文理研究大学高等师范学院）
Iteratively reweighted least square (IRLS) is a popular approach to solve sparsity-enforcing regression problems in machine learning. State of the art approaches are more efficient but typically rely on specific coordinate pruning schemes. In this work, we show how a surprisingly simple reparametrization of IRLS, coupled with a bilevel resolution (instead of an alternating scheme) is able to achieve top performances on a wide range of sparsity (such as Lasso, group Lasso and trace norm regularizations), regularization strength (including hard constraints), and design matrices (ranging from cor…
This review discusses methods for learning parameters for image reconstruction problems using bilevel formulations. Image reconstruction typically involves optimizing a cost function to recover a vector of unknown variables that agrees with collected measurements and prior assumptions. State-of-the-art image reconstruction methods learn these prior assumptions from training data using various machine learning techniques, such as bilevel methods. One can view the bilevel problem as formalizing hyperparameter optimization, as bridging machine learning and cost function based optimization methods…
Simple bilevel problems are optimization problems in which we want to find an optimal solution to an inner problem that minimizes an outer objective function. Such problems appear in many machine learning and signal processing applications as a way to eliminate undesirable solutions. %However, since these problems do not satisfy regularity conditions, they are often hard to solve exactly and are usually solved via iterative regularization. In the past few years, several algorithms were proposed to solve these bilevel problems directly and provide a rate for obtaining feasibility, assuming that…
We consider a bilevel program involving a linear lower level problem with left-hand-side perturbation. We then consider the Karush-Kuhn-Tucker reformulation of the problem and subsequently build a tractable optimization problem with linear constraints by means of a partial exact penalization. A semismooth system of equations is then generated from the later problem and a Newton-type method is developed to solve it. Finally, we illustrate the convergence and practical implementation of the algorithm on the optimal toll-setting problem in transportation networks.
In this paper, we introduce an inertial proximal method for solving a bilevel problem involving two monotone equilibrium bifunctions in Hilbert spaces. Under suitable conditions and without any restrictive assumption on the trajectories, the weak and strong convergence of the sequence generated by the iterative method are established. Two particular cases illustrating the proposed method are thereafter discussed with respect to hierarchical minimization problems and equilibrium problems under saddle point constraint. Furthermore, a numerical example is given to demonstrate the implementability…
Partial calmness is a celebrated but restrictive property of bilevel optimization problems whose presence opens a way to the derivation of Karush--Kuhn--Tucker-type necessary optimality conditions in order to characterize local minimizers. In the past, sufficient conditions for the validity of partial calmness have been investigated. In this regard, the presence of a linearly structured lower level problem has turned out to be beneficial. However, the associated literature suffers from inaccurate results. In this note, we clarify some regarding erroneous statements and visualize the underlying…
Support vector classification (SVC) is a classical and well-performed learning method for classification problems. A regularization parameter, which significantly affects the classification performance, has to be chosen and this is usually done by the cross-validation procedure. In this paper, we reformulate the hyperparameter selection problem for support vector classification as a bilevel optimization problem in which the upper-level problem minimizes the average number of misclassified data points over all the cross-validation folds, and the lower-level problems are the l1-loss SVC problems…
University of Pittsburgh（匹兹堡大学）；Nanjing University of Aeronautics & Astronautics（南京航空航天大学）
Bilevel optimization has been recently used in many machine learning problems such as hyperparameter optimization, policy optimization, and meta learning. Although many bilevel optimization methods have been proposed, they still suffer from the high computational complexities and do not consider the more general bilevel problems with nonsmooth regularization. In the paper, thus, we propose a class of enhanced bilevel optimization methods with using Bregman distance to solve bilevel optimization problems, where the outer subproblem is nonconvex and possibly nonsmooth, and the inner subproblem i…
Technical University of Munich（慕尼黑工业大学）；Munich Data Science Institute（慕尼黑数据科学研究所）；Polytechnique Montréal（蒙特利尔理工大学）；Pontifical Catholic University of Rio de Janeiro（里约热内卢天主教大学）
We study the feature-based newsvendor problem, in which a decision-maker has access to historical data consisting of demand observations and exogenous features. In this setting, we investigate feature selection, aiming to derive sparse, explainable models with improved out-of-sample performance. Up to now, state-of-the-art methods utilize regularization, which penalizes the number of selected features or the norm of the solution vector. As an alternative, we introduce a novel bilevel programming formulation. The upper-level problem selects a subset of features that minimizes an estimate of the…
We propose a bilevel optimization approach for the estimation of parameters in nonlocal image denoising models. The parameters we consider are both the fidelity weight and weights within the kernel of the nonlocal operator. In both cases we investigate the differentiability of the solution operator in function spaces and derive a first order optimality system that characterizes local minima. For the numerical solution of the problems, we use a second-order trust-region algorithm in combination with a finite element discretization of the nonlocal denoising models and we introduce a computationa…
One fundamental problem when solving inverse problems is how to find regularization parameters. This article considers solving this problem using data-driven bilevel optimization, i.e. we consider the adaptive learning of the regularization parameter from data by means of optimization. This approach can be interpreted as solving an empirical risk minimization problem, and we analyze its performance in the large data sample size limit for general nonlinear problems. We demonstrate how to implement our framework on linear inverse problems, where we can further show the inverse accuracy does not …
Shandong University（山东大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；Huawei（华为）
The (gradient-based) bilevel programming framework is widely used in hyperparameter optimization and has achieved excellent performance empirically. Previous theoretical work mainly focuses on its optimization properties, while leaving the analysis on generalization largely open. This paper attempts to address the issue by presenting an expectation bound w.r.t. the validation set based on uniform stability. Our results can explain some mysterious behaviours of the bilevel programming in practice, for instance, overfitting to the validation set. We also present an expectation bound for the clas…
This article concerns an optimal crowd motion control problem in which the crowd features a structure given by its organization into N groups (participants) each one spatially confined in a set. The overall optimal control problem consists in driving the ensemble of sets as close as possible to a given point (the 'exit') while the population in each set minimizes its control effort subject to its sweeping dynamics with a controlled state dependent velocity drift. In order to capture the conflict between the goal of the overall population and those of the various groups, the problem is cast as …
