As the title suggests, this is the third paper in a series addressing bilevel optimization problems that are governed by the Kantorovich problem of optimal transport. These tasks can be reformulated as mathematical problems with complementarity constraints in the space of regular Borel measures. Due to the nonsmoothness that is introduced by the complementarity constraints, such problems are often regularized, for instance, using entropic regularization. In this series of papers, however, we apply a quadratic regularization to the Kantorovich problem. By doing so, we enhance its numerical properties while preserving the sparsity structure of the optimal transportation plan as much as possible. While the first two papers in this series focus on the well-posedness of the regularized bilevel problems and the approximation of solutions to the bilevel optimization problem in the infinite-dimensional case, in this paper, we reproduce these results for the finite-dimensional case and present findings that go well beyond the ones of the previous papers and pave the way for the numerical treatment of the bilevel problems.
This paper is concerned with an optimization problem that is constrained by the Kantorovich optimal transportation problem. This bilevel optimization problem can be reformulated as a mathematical problem with complementarity constraints in the space of regular Borel measures. Because of the non-smoothness induced by the complementarity relations, problems of this type are frequently regularized. Here we apply a quadratic regularization of the Kantorovich problem. As the title indicates, this is the second part in a series of three papers. While the existence of optimal solutions to both the bi…
This paper is concerned with an optimization problem governed by the Kantorovich optimal transportation problem. This gives rise to a bilevel optimization problem, which can be reformulated as a mathematical problem with complementarity constraints in the space of regular Borel measures. Because of the non-smoothness induced by the complementarity relations, problems of this type are frequently regularized. Here we apply a quadratic regularization of the Kantorovich problem. As the title indicates, this is the first part in a series of three papers. It addresses the existence of optimal soluti…
We address the problem of optimal scale-dependent parameter learning in total variation image denoising. Such problems are formulated as bilevel optimization instances with total variation denoising problems as lower-level constraints. For the bilevel problem, we are able to derive M-stationarity conditions, after characterizing the corresponding Mordukhovich generalized normal cone and verifying suitable constraint qualification conditions. We also derive B-stationarity conditions, after investigating the Lipschitz continuity and directional differentiability of the lower-level solution opera…
A learning approach to selecting regularization parameters in multi-penalty Tikhonov regularization is investigated. It leads to a bilevel optimization problem, where the lower level problem is a Tikhonov regularized problem parameterized in the regularization parameters. Conditions which ensure the existence of solutions to the bilevel optimization problem of interest are derived, and these conditions are verified for two relevant examples. Difficulties arising from the possible lack of convexity of the lower level problems are discussed. Optimality conditions are given provided that a reason…
Total Generalized Variation (TGV) regularization in image reconstruction relies on an infimal convolution type combination of generalized first- and second-order derivatives. This helps to avoid the staircasing effect of Total Variation (TV) regularization, while still preserving sharp contrasts in images. The associated regularization effect crucially hinges on two parameters whose proper adjustment represents a challenging task. In this work, a bilevel optimization framework with a suitable statistics-based upper level objective is proposed in order to automatically select these parameters. …
We consider a bilevel optimization approach in function space for the choice of spatially dependent regularization parameters in TV image restoration models. First- and second-order optimality conditions for the bilevel problem are studied, when the spatially-dependent parameter belongs to the Sobolev space $H^1(Ω)$. A combined Schwarz domain decomposition-semismooth Newton method is proposed for the solution of the full optimality system and local superlinear convergence of the semismooth Newton method is analyzed. Exhaustive numerical computations are finally carried out to show the suitabil…
We consider bilevel linear problems, where some parameters are stochastic, and the leader has to decide in a here-and-now fashion, while the follower has complete information. In this setting, the leader's outcome can be modeled by a random variable, which we evaluate based on some law-invariant convex risk measure. A qualitative stability result under perturbations of the underlying probability distribution is presented. Moreover, for the expectation, the expected excess, and the upper semideviation, we establish Lipschitz continuity as well as sufficient conditions for differentiability. Fin…
In this paper, we exploit the so-called value function reformulation of the bilevel optimization problem to develop duality results for the problem. Our approach builds on Fenchel-Lagrange-type duality to establish suitable results for the bilevel optimization problem. First, we overview some standard duality results to show that they are not applicable to our problem. Secondly, via the concept of partial calmness, we establish weak and strong duality results. In particular, Lagrange, Fenchel-Lagrange, and Toland-Fenchel- Lagrange duality concepts are investigated for this type of problems und…
Bilevel programs are optimization problems where some variables are solutions to optimization problems themselves, and they arise in a variety of control applications, including: control of vehicle traffic networks, inverse reinforcement learning and inverse optimization, and robust control for human-automation systems. This paper develops a duality-based approach to solving bilevel programs where the lower level problem is convex. Our approach is to use partial dualization to construct a new dual function that is differentiable, unlike the Lagrangian dual that is only directionally differenti…
George Mason University（乔治梅森大学）；Argonne National Laboratory（阿贡国家实验室）
In this work we consider a generalized bilevel optimization framework for solving inverse problems. We introduce fractional Laplacian as a regularizer to improve the reconstruction quality, and compare it with the total variation regularization. We emphasize that the key advantage of using fractional Laplacian as a regularizer is that it leads to a linear operator, as opposed to the total variation regularization which results in a nonlinear degenerate operator. Inspired by residual neural networks, to learn the optimal strength of regularization and the exponent of fractional Laplacian, we de…
In this work we analyze regularized optimal transport problems in the so-called Kantorovich form, i.e. given two Radon measures on two compact sets, the aim is to find a transport plan, which is another Radon measure on the product of the sets, that has these two measures as marginals and minimizes the sum of a certain linear cost function and a regularization term. We focus on regularization terms where a Young's function applied to the (density of the) transport plan is integrated against a product measure. This forces the transport plan to belong to a certain Orlicz space. The predual probl…
Motivated by emerging applications in wireless sensor networks and large-scale data processing, we consider distributed optimization over directed networks where the agents communicate their information locally to their neighbors to cooperatively minimize a global cost function. We introduce a new unifying distributed constrained optimization model that is characterized as a bilevel optimization problem. This model captures a wide range of existing problems over directed networks including: (i) Distributed optimization with linear constraints; (ii) Distributed unconstrained nonstrongly convex …
Whiteness-based bilevel learning of regularization parameters in imaging
University of Insubria（因苏布里亚大学）；Laboratoire I3S（I3S实验室）；CNRS（法国国家科学研究中心）；UniCA（尼斯蔚蓝海岸大学）；Inria（法国国家信息与自动化研究所）；University of Naples Federico II（那不勒斯费德里科二世大学）；University of Bologna（博洛尼亚大学）
A Single-Loop Gradient Algorithm for Pessimistic Bilevel Optimization via Smooth Approximation
Southern University of Science and Technology（南方科技大学）；National Center for Applied Mathematics Shenzhen, Southern University of Science and Technology（深圳应用数学国家中心）；Detection Institute for Advanced Technology Longhua-Shenzhen (DIATLHSZ)（龙华-深圳先进技术检测研究院）
We analyze continuous optimal transport problems in the so-called Kantorovich form, where we seek a transport plan between two marginals that are probability measures on compact subsets of Euclidean space. We consider the case of regularization with the negative entropy with respect to the Lebesgue measure, which has attracted attention because it can be solved by the very simple Sinkhorn algorithm. We first analyze the regularized problem in the context of classical Fenchel duality and derive a strong duality result for a predual problem in the space of continuous functions. However, this pro…
This review discusses methods for learning parameters for image reconstruction problems using bilevel formulations. Image reconstruction typically involves optimizing a cost function to recover a vector of unknown variables that agrees with collected measurements and prior assumptions. State-of-the-art image reconstruction methods learn these prior assumptions from training data using various machine learning techniques, such as bilevel methods. One can view the bilevel problem as formalizing hyperparameter optimization, as bridging machine learning and cost function based optimization methods…
University of Bath（巴斯大学）；CNRS（法国国家科学研究中心）；Ecole Normale Supérieure, PSL University（巴黎文理研究大学高等师范学院）
Iteratively reweighted least square (IRLS) is a popular approach to solve sparsity-enforcing regression problems in machine learning. State of the art approaches are more efficient but typically rely on specific coordinate pruning schemes. In this work, we show how a surprisingly simple reparametrization of IRLS, coupled with a bilevel resolution (instead of an alternating scheme) is able to achieve top performances on a wide range of sparsity (such as Lasso, group Lasso and trace norm regularizations), regularization strength (including hard constraints), and design matrices (ranging from cor…
Simple bilevel problems are optimization problems in which we want to find an optimal solution to an inner problem that minimizes an outer objective function. Such problems appear in many machine learning and signal processing applications as a way to eliminate undesirable solutions. %However, since these problems do not satisfy regularity conditions, they are often hard to solve exactly and are usually solved via iterative regularization. In the past few years, several algorithms were proposed to solve these bilevel problems directly and provide a rate for obtaining feasibility, assuming that…
A learning approach for determining which operator from a class of nonlocal operators is optimal for the regularization of an inverse problem is investigated. The considered class of nonlocal operators is motivated by the use of squared fractional order Sobolev seminorms as regularization operators. First fundamental results from the theory of regularization with local operators are extended to the nonlocal case. Then a framework based on a bilevel optimization strategy is developed which allows to choose nonlocal regularization operators from a given class which i) are optimal with respect to…
We study a class of bilevel convex optimization problems where the goal is to find the minimizer of an objective function in the upper level, among the set of all optimal solutions of an optimization problem in the lower level. A wide range of problems in convex optimization can be formulated using this class. An important example is the case where an optimization problem is ill-posed. In this paper, our interest lies in addressing the bilevel problems, where the lower level objective is given as a finite sum of separate nondifferentiable convex component functions. This is the case in a varie…
Support vector classification (SVC) is a classical and well-performed learning method for classification problems. A regularization parameter, which significantly affects the classification performance, has to be chosen and this is usually done by the cross-validation procedure. In this paper, we reformulate the hyperparameter selection problem for support vector classification as a bilevel optimization problem in which the upper-level problem minimizes the average number of misclassified data points over all the cross-validation folds, and the lower-level problems are the l1-loss SVC problems…
Motivated by high-dimensional nonlinear optimization problems as well as ill-posed optimization problems arising in image processing, we consider a bilevel optimization model where we seek among the optimal solutions of the inner level problem, a solution that minimizes a secondary metric. Our goal is to address the high-dimensionality of the bilevel problem, and the nondifferentiability of the objective function. Minimal norm gradient, sequential averaging, and iterative regularization are some of the recent schemes developed for addressing the bilevel problem. But none of them address the hi…
We investigate the continuous optimal transport problem in the so-called Kantorovich form, i.e. given two Radon measures on two compact sets, we seek an optimal transport plan which is another Radon measure on the product of the sets that has these two measures as marginals and minimizes a certain cost function. We consider regularization of the problem with so-called Young's functions, which forces the optimal transport plan to be a function in the corresponding Orlicz space rather than a Radon measure. We derive the predual problem and show strong duality and existence of primal solutions to…
University of Pittsburgh（匹兹堡大学）；Nanjing University of Aeronautics & Astronautics（南京航空航天大学）
Bilevel optimization has been recently used in many machine learning problems such as hyperparameter optimization, policy optimization, and meta learning. Although many bilevel optimization methods have been proposed, they still suffer from the high computational complexities and do not consider the more general bilevel problems with nonsmooth regularization. In the paper, thus, we propose a class of enhanced bilevel optimization methods with using Bregman distance to solve bilevel optimization problems, where the outer subproblem is nonconvex and possibly nonsmooth, and the inner subproblem i…
Technical University of Munich（慕尼黑工业大学）；Munich Data Science Institute（慕尼黑数据科学研究所）；Polytechnique Montréal（蒙特利尔理工大学）；Pontifical Catholic University of Rio de Janeiro（里约热内卢天主教大学）
We study the feature-based newsvendor problem, in which a decision-maker has access to historical data consisting of demand observations and exogenous features. In this setting, we investigate feature selection, aiming to derive sparse, explainable models with improved out-of-sample performance. Up to now, state-of-the-art methods utilize regularization, which penalizes the number of selected features or the norm of the solution vector. As an alternative, we introduce a novel bilevel programming formulation. The upper-level problem selects a subset of features that minimizes an estimate of the…
Shandong University（山东大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；Huawei（华为）
The (gradient-based) bilevel programming framework is widely used in hyperparameter optimization and has achieved excellent performance empirically. Previous theoretical work mainly focuses on its optimization properties, while leaving the analysis on generalization largely open. This paper attempts to address the issue by presenting an expectation bound w.r.t. the validation set based on uniform stability. Our results can explain some mysterious behaviours of the bilevel programming in practice, for instance, overfitting to the validation set. We also present an expectation bound for the clas…
One fundamental problem when solving inverse problems is how to find regularization parameters. This article considers solving this problem using data-driven bilevel optimization, i.e. we consider the adaptive learning of the regularization parameter from data by means of optimization. This approach can be interpreted as solving an empirical risk minimization problem, and we analyze its performance in the large data sample size limit for general nonlinear problems. We demonstrate how to implement our framework on linear inverse problems, where we can further show the inverse accuracy does not …
We study the potential functions that determine the optimal density for $\varepsilon$-entropically regularized optimal transport, the so-called Schrödinger potentials, and their convergence to the counterparts in classical optimal transport, the Kantorovich potentials. In the limit $\varepsilon\to0$ of vanishing regularization, strong compactness holds in $L^{1}$ and cluster points are Kantorovich potentials. In particular, the Schrödinger potentials converge in $L^{1}$ to the Kantorovich potentials as soon as the latter are unique. These results are proved for all continuous, integrable cost …
An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning
EPSRC Centre for Doctoral Training in Statistical Applied Mathematics at Bath (SAMBa)（EPSRC统计应用数学博士培训中心（SAMBa））；Department of Mathematical Sciences, University of Bath（巴斯大学数学科学系）；Department of Electronics & Electrical Communication Engineering, Indian Institute of Technology (IIT), Kharagpur（印度理工学院（Kharagpur）电子与电气通信工程系）；School of Mathematics and Statistics, University of Sydney（悉尼大学数学与统计学学院）
We introduce and study a mathematical framework for a broad class of regularization functionals for ill-posed inverse problems: Regularization Graphs. Regularization graphs allow to construct functionals using as building blocks linear operators and convex functionals, assembled by means of operators that can be seen as generalizations of classical infimal convolution operators. This class of functionals exhaustively covers existing regularization approaches and it is flexible enough to craft new ones in a simple and constructive way. We provide well-posedness and convergence results with the …
