As the title suggests, this is the third paper in a series addressing bilevel optimization problems that are governed by the Kantorovich problem of optimal transport. These tasks can be reformulated as mathematical problems with complementarity constraints in the space of regular Borel measures. Due to the nonsmoothness that is introduced by the complementarity constraints, such problems are often regularized, for instance, using entropic regularization. In this series of papers, however, we apply a quadratic regularization to the Kantorovich problem. By doing so, we enhance its numerical properties while preserving the sparsity structure of the optimal transportation plan as much as possible. While the first two papers in this series focus on the well-posedness of the regularized bilevel problems and the approximation of solutions to the bilevel optimization problem in the infinite-dimensional case, in this paper, we reproduce these results for the finite-dimensional case and present findings that go well beyond the ones of the previous papers and pave the way for the numerical treatment of the bilevel problems.
This paper is concerned with an optimization problem that is constrained by the Kantorovich optimal transportation problem. This bilevel optimization problem can be reformulated as a mathematical problem with complementarity constraints in the space of regular Borel measures. Because of the non-smoothness induced by the complementarity relations, problems of this type are frequently regularized. Here we apply a quadratic regularization of the Kantorovich problem. As the title indicates, this is the second part in a series of three papers. While the existence of optimal solutions to both the bi…
This paper is concerned with an optimization problem governed by the Kantorovich optimal transportation problem. This gives rise to a bilevel optimization problem, which can be reformulated as a mathematical problem with complementarity constraints in the space of regular Borel measures. Because of the non-smoothness induced by the complementarity relations, problems of this type are frequently regularized. Here we apply a quadratic regularization of the Kantorovich problem. As the title indicates, this is the first part in a series of three papers. It addresses the existence of optimal soluti…
We address the problem of optimal scale-dependent parameter learning in total variation image denoising. Such problems are formulated as bilevel optimization instances with total variation denoising problems as lower-level constraints. For the bilevel problem, we are able to derive M-stationarity conditions, after characterizing the corresponding Mordukhovich generalized normal cone and verifying suitable constraint qualification conditions. We also derive B-stationarity conditions, after investigating the Lipschitz continuity and directional differentiability of the lower-level solution opera…
Total Generalized Variation (TGV) regularization in image reconstruction relies on an infimal convolution type combination of generalized first- and second-order derivatives. This helps to avoid the staircasing effect of Total Variation (TV) regularization, while still preserving sharp contrasts in images. The associated regularization effect crucially hinges on two parameters whose proper adjustment represents a challenging task. In this work, a bilevel optimization framework with a suitable statistics-based upper level objective is proposed in order to automatically select these parameters. …
In this paper, we exploit the so-called value function reformulation of the bilevel optimization problem to develop duality results for the problem. Our approach builds on Fenchel-Lagrange-type duality to establish suitable results for the bilevel optimization problem. First, we overview some standard duality results to show that they are not applicable to our problem. Secondly, via the concept of partial calmness, we establish weak and strong duality results. In particular, Lagrange, Fenchel-Lagrange, and Toland-Fenchel- Lagrange duality concepts are investigated for this type of problems und…
In this work we analyze regularized optimal transport problems in the so-called Kantorovich form, i.e. given two Radon measures on two compact sets, the aim is to find a transport plan, which is another Radon measure on the product of the sets, that has these two measures as marginals and minimizes the sum of a certain linear cost function and a regularization term. We focus on regularization terms where a Young's function applied to the (density of the) transport plan is integrated against a product measure. This forces the transport plan to belong to a certain Orlicz space. The predual probl…
Motivated by emerging applications in wireless sensor networks and large-scale data processing, we consider distributed optimization over directed networks where the agents communicate their information locally to their neighbors to cooperatively minimize a global cost function. We introduce a new unifying distributed constrained optimization model that is characterized as a bilevel optimization problem. This model captures a wide range of existing problems over directed networks including: (i) Distributed optimization with linear constraints; (ii) Distributed unconstrained nonstrongly convex …
Whiteness-based bilevel learning of regularization parameters in imaging
University of Insubria（因苏布里亚大学）；Laboratoire I3S（I3S实验室）；CNRS（法国国家科学研究中心）；UniCA（尼斯蔚蓝海岸大学）；Inria（法国国家信息与自动化研究所）；University of Naples Federico II（那不勒斯费德里科二世大学）；University of Bologna（博洛尼亚大学）
A Single-Loop Gradient Algorithm for Pessimistic Bilevel Optimization via Smooth Approximation
Southern University of Science and Technology（南方科技大学）；National Center for Applied Mathematics Shenzhen, Southern University of Science and Technology（深圳应用数学国家中心）；Detection Institute for Advanced Technology Longhua-Shenzhen (DIATLHSZ)（龙华-深圳先进技术检测研究院）
We analyze continuous optimal transport problems in the so-called Kantorovich form, where we seek a transport plan between two marginals that are probability measures on compact subsets of Euclidean space. We consider the case of regularization with the negative entropy with respect to the Lebesgue measure, which has attracted attention because it can be solved by the very simple Sinkhorn algorithm. We first analyze the regularized problem in the context of classical Fenchel duality and derive a strong duality result for a predual problem in the space of continuous functions. However, this pro…
This review discusses methods for learning parameters for image reconstruction problems using bilevel formulations. Image reconstruction typically involves optimizing a cost function to recover a vector of unknown variables that agrees with collected measurements and prior assumptions. State-of-the-art image reconstruction methods learn these prior assumptions from training data using various machine learning techniques, such as bilevel methods. One can view the bilevel problem as formalizing hyperparameter optimization, as bridging machine learning and cost function based optimization methods…
University of Bath（巴斯大学）；CNRS（法国国家科学研究中心）；Ecole Normale Supérieure, PSL University（巴黎文理研究大学高等师范学院）
Iteratively reweighted least square (IRLS) is a popular approach to solve sparsity-enforcing regression problems in machine learning. State of the art approaches are more efficient but typically rely on specific coordinate pruning schemes. In this work, we show how a surprisingly simple reparametrization of IRLS, coupled with a bilevel resolution (instead of an alternating scheme) is able to achieve top performances on a wide range of sparsity (such as Lasso, group Lasso and trace norm regularizations), regularization strength (including hard constraints), and design matrices (ranging from cor…
Simple bilevel problems are optimization problems in which we want to find an optimal solution to an inner problem that minimizes an outer objective function. Such problems appear in many machine learning and signal processing applications as a way to eliminate undesirable solutions. %However, since these problems do not satisfy regularity conditions, they are often hard to solve exactly and are usually solved via iterative regularization. In the past few years, several algorithms were proposed to solve these bilevel problems directly and provide a rate for obtaining feasibility, assuming that…
Support vector classification (SVC) is a classical and well-performed learning method for classification problems. A regularization parameter, which significantly affects the classification performance, has to be chosen and this is usually done by the cross-validation procedure. In this paper, we reformulate the hyperparameter selection problem for support vector classification as a bilevel optimization problem in which the upper-level problem minimizes the average number of misclassified data points over all the cross-validation folds, and the lower-level problems are the l1-loss SVC problems…
University of Pittsburgh（匹兹堡大学）；Nanjing University of Aeronautics & Astronautics（南京航空航天大学）
Bilevel optimization has been recently used in many machine learning problems such as hyperparameter optimization, policy optimization, and meta learning. Although many bilevel optimization methods have been proposed, they still suffer from the high computational complexities and do not consider the more general bilevel problems with nonsmooth regularization. In the paper, thus, we propose a class of enhanced bilevel optimization methods with using Bregman distance to solve bilevel optimization problems, where the outer subproblem is nonconvex and possibly nonsmooth, and the inner subproblem i…
Technical University of Munich（慕尼黑工业大学）；Munich Data Science Institute（慕尼黑数据科学研究所）；Polytechnique Montréal（蒙特利尔理工大学）；Pontifical Catholic University of Rio de Janeiro（里约热内卢天主教大学）
We study the feature-based newsvendor problem, in which a decision-maker has access to historical data consisting of demand observations and exogenous features. In this setting, we investigate feature selection, aiming to derive sparse, explainable models with improved out-of-sample performance. Up to now, state-of-the-art methods utilize regularization, which penalizes the number of selected features or the norm of the solution vector. As an alternative, we introduce a novel bilevel programming formulation. The upper-level problem selects a subset of features that minimizes an estimate of the…
Shandong University（山东大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；Huawei（华为）
The (gradient-based) bilevel programming framework is widely used in hyperparameter optimization and has achieved excellent performance empirically. Previous theoretical work mainly focuses on its optimization properties, while leaving the analysis on generalization largely open. This paper attempts to address the issue by presenting an expectation bound w.r.t. the validation set based on uniform stability. Our results can explain some mysterious behaviours of the bilevel programming in practice, for instance, overfitting to the validation set. We also present an expectation bound for the clas…
One fundamental problem when solving inverse problems is how to find regularization parameters. This article considers solving this problem using data-driven bilevel optimization, i.e. we consider the adaptive learning of the regularization parameter from data by means of optimization. This approach can be interpreted as solving an empirical risk minimization problem, and we analyze its performance in the large data sample size limit for general nonlinear problems. We demonstrate how to implement our framework on linear inverse problems, where we can further show the inverse accuracy does not …
We study the potential functions that determine the optimal density for $\varepsilon$-entropically regularized optimal transport, the so-called Schrödinger potentials, and their convergence to the counterparts in classical optimal transport, the Kantorovich potentials. In the limit $\varepsilon\to0$ of vanishing regularization, strong compactness holds in $L^{1}$ and cluster points are Kantorovich potentials. In particular, the Schrödinger potentials converge in $L^{1}$ to the Kantorovich potentials as soon as the latter are unique. These results are proved for all continuous, integrable cost …
An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning
EPSRC Centre for Doctoral Training in Statistical Applied Mathematics at Bath (SAMBa)（EPSRC统计应用数学博士培训中心（SAMBa））；Department of Mathematical Sciences, University of Bath（巴斯大学数学科学系）；Department of Electronics & Electrical Communication Engineering, Indian Institute of Technology (IIT), Kharagpur（印度理工学院（Kharagpur）电子与电气通信工程系）；School of Mathematics and Statistics, University of Sydney（悉尼大学数学与统计学学院）
We introduce and study a mathematical framework for a broad class of regularization functionals for ill-posed inverse problems: Regularization Graphs. Regularization graphs allow to construct functionals using as building blocks linear operators and convex functionals, assembled by means of operators that can be seen as generalizations of classical infimal convolution operators. This class of functionals exhaustively covers existing regularization approaches and it is flexible enough to craft new ones in a simple and constructive way. We provide well-posedness and convergence results with the …
