Distributionally robust optimization (DRO) is a widely used framework for optimizing objective functionals in the presence of both randomness and model-form uncertainty. A key step in the practical solution of many DRO problems is a tractable reformulation of the optimization over the chosen model ambiguity set, which is generally infinite dimensional. Previous works have solved this problem in the case where the objective functional is an expected value. In this paper we study objective functionals that are the sum of an expected value and a variance penalty term. We prove that the corresponding variance-penalized DRO problem over an $f$-divergence neighborhood can be reformulated as a finite-dimensional convex optimization problem. This result also provides tight uncertainty quantification bounds on the variance.
The stochastic variational inequality problem (SVIP) is an equilibrium model that includes random variables and has been widely applied in various fields such as economics and engineering. Expected residual minimization (ERM) is an established model for obtaining a reasonable solution for the SVIP, and its objective function is an expected value of a suitable merit function for the SVIP. However, the ERM is restricted to the case where the distribution is known in advance. We extend the ERM to ensure the attainment of robust solutions for the SVIP under the uncertainty distribution (the extend…
We consider a nonlinear random walk which, in each time step, is free to choose its own transition probability within a neighborhood (w.r.t. Wasserstein distance) of the transition probability of a fixed Lévy process. In analogy to the classical framework we show that, when passing from discrete to continuous time via a scaling limit, this nonlinear random walk gives rise to a nonlinear semigroup. We explicitly compute the generator of this semigroup and corresponding PDE as a perturbation of the generator of the initial Lévy process.
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；The Hong Kong Polytechnic University（香港理工大学）；Imperial College Business School（帝国理工商学院）
The likelihood function is a fundamental component in Bayesian statistics. However, evaluating the likelihood of an observation is computationally intractable in many applications. In this paper, we propose a non-parametric approximation of the likelihood that identifies a probability measure which lies in the neighborhood of the nominal measure and that maximizes the probability of observing the given sample point. We show that when the neighborhood is constructed by the Kullback-Leibler divergence, by moment conditions or by the Wasserstein distance, then our \textit{optimistic likelihood} c…
University College London（伦敦大学学院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
In this paper we explore the relation between distributionally robust learning and different forms of regularization to enforce robustness of deep neural networks. In particular, starting from a concrete min-max distributionally robust problem, and using tools from optimal transport theory, we derive first order and second order approximations to the distributionally robust problem in terms of appropriate regularized risk minimization problems. In the context of deep ResNet models, we identify the structure of the resulting regularization problems as mean-field optimal control problems where t…
Established approaches to obtain generalization bounds in data-driven optimization and machine learning mostly build on solutions from empirical risk minimization (ERM), which depend crucially on the functional complexity of the hypothesis class. In this paper, we present an alternate route to obtain these bounds on the solution from distributionally robust optimization (DRO), a recent data-driven optimization framework based on worst-case analysis and the notion of ambiguity set to capture statistical uncertainty. In contrast to the hypothesis class complexity in ERM, our DRO bounds depend on…
Distributionally robust optimization (DRO) is a worst-case framework for stochastic optimization under uncertainty that has drawn fast-growing studies in recent years. When the underlying probability distribution is unknown and observed from data, DRO suggests to compute the worst-case distribution within a so-called uncertainty set that captures the involved statistical uncertainty. In particular, DRO with uncertainty set constructed as a statistical divergence neighborhood ball has been shown to provide a tool for constructing valid confidence intervals for nonparametric functionals, and bea…
IBM Research AI（IBM研究院人工智能中心）；Colorado State University-Pueblo（科罗拉多州立大学普韦布洛分校）
Distributionally robust optimization (DRO) problems are increasingly seen as a viable method to train machine learning models for improved model generalization. These min-max formulations, however, are more difficult to solve. We therefore provide a new stochastic gradient descent algorithm to efficiently solve this DRO formulation. Our approach applies gradient descent to the outer minimization formulation and estimates the gradient of the inner maximization based on a sample average approximation. The latter uses a subset of the data in each iteration, progressively increasing the subset siz…
We consider a distributionally robust formulation of stochastic optimization problems arising in statistical learning, where robustness is with respect to uncertainty in the underlying data distribution. Our formulation builds on risk-averse optimization techniques and the theory of coherent risk measures. It uses semi-deviation risk for quantifying uncertainty, allowing us to compute solutions that are robust against perturbations in the population data distribution. We consider a large family of loss functions that can be non-convex and non-smooth and develop an efficient stochastic subgradi…
This paper studies a structured compound stochastic program (SP) involving multiple expectations coupled by nonconvex and nonsmooth functions. We present a successive convex-programming based sampling algorithm and establish its subsequential convergence. We describe stationarity properties of the limit points for several classes of the compound SP. We further discuss probabilistic stopping rules based on the computable error-bound for the algorithm. We present several risk measure minimization problems that can be formulated as such a compound stochastic program; these include generalized dev…
Wasserstein Distributionally Robust Optimization with Heterogeneous Data Sources
Risk Analytics and Optimization Laboratory, École polytechnique fédérale de Lausanne（洛桑联邦理工学院风险分析与优化实验室）；Department of Computer and Systems Sciences & Digital Futures, Stockholm University（斯德哥尔摩大学计算机与系统科学及数字未来系）；Department of Mathematical Analysis, Statistics and Operations Research and Applied Mathematics (Area of Statistics and Operations Research), University of Málaga（马拉加大学数学分析、统计与运筹学及应用数学系（统计与运筹学领域））
A stochastic smoothing framework for nonconvex-nonconcave minEmax problems with applications to Wasserstein distributionally robust optimization
Department of Applied Mathematics, Hong Kong Polytechnic University, Hong Kong, China（应用数学系，香港理工大学）；Institute for Math & AI, Wuhan (IMAI), Wuhan university, China（数学与人工智能研究院，武汉大学）；Department of Mathematical Sciences Rensselaer Polytechnic Institute, Troy, NY, USA（数学科学系，伦斯勒理工学院）
University of California, Berkeley（加州大学伯克利分校）；University of Washington, Seattle（西雅图华盛顿大学）
Min-max optimization is emerging as a key framework for analyzing problems of robustness to strategically and adversarially generated data. We propose a random reshuffling-based gradient free Optimistic Gradient Descent-Ascent algorithm for solving convex-concave min-max problems with finite sum structure. We prove that the algorithm enjoys the same convergence rate as that of zeroth-order algorithms for convex minimization problems. We further specialize the algorithm to solve distributionally robust, decision-dependent learning problems, where gradient information is not readily available. T…
Stanford University（斯坦福大学）；Chinese University of Hong Kong（香港中文大学）
Distributionally robust optimization has been shown to offer a principled way to regularize learning models. In this paper, we find that Tikhonov regularization is distributionally robust in an optimal transport sense (i.e., if an adversary chooses distributions in a suitable optimal transport neighborhood of the empirical measure), provided that suitable martingale constraints are also imposed. Further, we introduce a relaxation of the martingale constraints which not only provides a unified viewpoint to a class of existing robust methods but also leads to new regularization tools. To realize…
The University of Sydney（悉尼大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）
We study a model for adversarial classification based on distributionally robust chance constraints. We show that under Wasserstein ambiguity, the model aims to minimize the conditional value-at-risk of the distance to misclassification, and we explore links to adversarial classification models proposed earlier and to maximum-margin classifiers. We also provide a reformulation of the distributionally robust model for linear classification, and show it is equivalent to minimizing a regularized ramp loss objective. Numerical experiments show that, despite the nonconvexity of this formulation, st…
A local convergence result for abstract descent methods is proved. The sequence of iterates is attracted by a local (or global) minimum, stays in its neighborhood and converges within this neighborhood. This result allows algorithms to exploit local properties of the objective function. In particular, the abstract theory in this paper applies to the inertial forward--backward splitting method: iPiano---a generalization of the Heavy-ball method. Moreover, it reveals an equivalence between iPiano and inertial averaged/alternating proximal minimization and projection methods. Key for this equival…
We study a tight Bennett-type concentration inequality for sums of heterogeneous and independent variables, defined as a one-dimensional minimization. We show that this refinement, which outperforms the standard known bounds, remains computationally tractable: we develop a polynomial-time algorithm to compute confidence bounds, proved to terminate with an epsilon-solution. From the proposed inequality, we deduce tight distributionally robust bounds to Chance-Constrained Programming problems. To illustrate the efficiency of our approach, we consider two use cases. First, we study the chance-con…
The problem of interest is the minimization of a nonlinear function subject to nonlinear equality constraints using a sequential quadratic programming (SQP) method. The minimization must be performed while observing only noisy evaluations of the objective and constraint functions. In order to obtain stability, the classical SQP method is modified by relaxing the standard Armijo line search based on the noise level in the functions, which is assumed to be known. Convergence theory is presented giving conditions under which the iterates converge to a neighborhood of the solution characterized by…
Enhancing Distributional Robustness in Principal Component Analysis by Wasserstein Distances
The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）；Academy of Mathematics and Systems Science（数学系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）
This note is concerned with the nonlinear Ky Fan $k$-norm matrix conic optimization problems, which include the nuclear norm regularized minimization problem as a special case. For this class of nonpolyhedral matrix conic optimization problems, under the assumption that a stationary solution satisfies the second-order sufficient condition and the associated Lagrange multiplier satisfies the strict Robinson's CQ, we show that two classes of perturbed KKT systems are locally upper Lipschitz at the origin, which implies a local error bound for the distance from any point in a neighborhood of the …
Beijing Institute of Technology（北京理工大学）；Beijing Institute of Technology Chongqing Innovation Center（北京理工大学重庆创新中心）；Tongji University（同济大学）；Nanyang Technological University（南洋理工大学）
The non-smooth finite-sum minimization is a fundamental problem in machine learning. This paper develops a distributed stochastic proximal-gradient algorithm with random reshuffling to solve the finite-sum minimization over time-varying multi-agent networks. The objective function is a sum of differentiable convex functions and non-smooth regularization. Each agent in the network updates local variables with a constant step-size by local information and cooperates to seek an optimal solution. We prove that local variable estimates generated by the proposed algorithm achieve consensus and are a…
Rutgers Business School（罗格斯商学院）；Penn State University（宾夕法尼亚州立大学）
In this work, we consider strongly convex strongly concave (SCSC) saddle point (SP) problems $\min_{x\in\mathbb{R}^{d_x}}\max_{y\in\mathbb{R}^{d_y}}f(x,y)$ where $f$ is $L$-smooth, $f(.,y)$ is $μ$-strongly convex for every $y$, and $f(x,.)$ is $μ$-strongly concave for every $x$. Such problems arise frequently in machine learning in the context of robust empirical risk minimization (ERM), e.g. $\textit{distributionally robust}$ ERM, where partial gradients are estimated using mini-batches of data points. Assuming we have access to an unbiased stochastic first-order oracle we consider the stocha…
Desautels Faculty of Management（德索特尔管理学院）；McGill University（麦吉尔大学）；GERAD（GERAD（蒙特利尔决策科学、运筹学和统计学研究组））；HEC Montréal（蒙特利尔高等商学院）；University of Cyprus（塞浦路斯大学）
