We propose an algorithm to solve optimization problems constrained by partial (ordinary) differential equations under uncertainty, with almost sure constraints on the state variable. To alleviate the computational burden of high-dimensional random variables, we approximate all random fields by the tensor-train decomposition. To enable efficient tensor-train approximation of the state constraints, the latter are handled using the Moreau-Yosida penalty, with an additional smoothing of the positive part (plus/ReLU) function by a softplus function. In a special case of a quadratic cost minimization constrained by linear elliptic partial differential equations, and some additional constraint qualification, we prove strong convergence of the regularized solution to the optimal control. This result also proposes a practical recipe for selecting the smoothing parameter as a function of the penalty parameter. We develop a second order Newton type method with a fast matrix-free action of the approximate Hessian to solve the smoothed Moreau-Yosida problem. This algorithm is tested on benchmark elliptic problems with random coefficients, optimization problems constrained by random elliptic variational inequalities, and a real-world epidemiological model with 20 random variables. These examples demonstrate mild (at most polynomial) scaling with respect to the dimension and regularization parameters.
In this paper, we introduce and study degenerate state-dependent sweeping processes with nonregular moving sets (subsmooth and positively $α$-far). Based on the Moreau-Yosida regularization, we prove the existence of solutions under the Lipschitzianity of the moving sets with respect to the truncated Hausdorff distance.
We analyze a potentially risk-averse convex stochastic optimization problem, where the control is deterministic and the state is a Banach-valued essentially bounded random variable. We obtain strong forms of necessary and sufficient optimality conditions for problems subject to equality and conical constraints. We propose a Moreau--Yosida regularization for the conical constraint and show consistency of the optimality conditions for the regularized problem as the regularization parameter is taken to infinity.
Alfréd Rényi Institute of Mathematics（阿尔弗雷德·雷尼数学研究所）
Variational representations of $f$-divergences are central to many machine learning algorithms, with Lipschitz constrained variants recently gaining attention. Inspired by this, we define the Moreau-Yosida approximation of $f$-divergences with respect to the Wasserstein-$1$ metric. The corresponding variational formulas provide a generalization of a number of recent results, novel special cases of interest and a relaxation of the hard Lipschitz constraint. Additionally, we prove that the so-called tight variational representation of $f$-divergences can be to be taken over the quotient space of…
These lecture notes for a graduate course cover generalized derivative concepts useful in deriving necessary optimality conditions and numerical algorithms for nondifferentiable optimization problems in inverse problems, imaging, and PDE-constrained optimization. Treated are convex functions and subdifferentials, Fenchel duality, monotone operators and resolvents, Moreau--Yosida regularization, proximal point and (some) first-order splitting methods, Clarke subdifferentials, and semismooth Newton methods. The required background from functional analysis and calculus of variations is also brief…
Proximal-Based Generative Modeling for Bayesian Inverse Problems
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉科学学院）；School of Mathematics, Sichuan University（四川大学数学学院）；School of Mathematical Sciences, Beijing University of Posts and Telecommunications（北京邮电大学数学科学学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Ministry of Education Key Laboratory of Mathematics and Information Networks（教育部数学与信息网络重点实验室）
In this paper we consider minimization of a difference-of-convex (DC) function with and without linear constraints. We first study a smooth approximation of a generic DC function, termed difference-of-Moreau-envelopes (DME) smoothing, where both components of the DC function are replaced by their respective Moreau envelopes. The resulting smooth approximation is shown to be Lipschitz differentiable, capture stationary points, local, and global minima of the original DC function, and enjoy some growth conditions, such as level-boundedness and coercivity, for broad classes of DC functions. We th…
Sample-Based Consistency in Infinite-Dimensional Conic-Constrained Stochastic Optimization
University of Klagenfurt（克拉根福大学）；H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院 H. 米尔顿·斯图尔特工业与系统工程学校）
We consider minimizing $f(x) = \mathbb{E}[f(x,ω)]$ when $f(x,ω)$ is possibly nonsmooth and either strongly convex or convex in $x$. (I) Strongly convex. When $f(x,ω)$ is $μ-$strongly convex in $x$, we propose a variable sample-size accelerated proximal scheme (VS-APM) and apply it on $f_η(x)$, the ($η$-)Moreau smoothed variant of $\mathbb{E}[f(x,ω)]$; we term such a scheme as (m-VS-APM). We consider three settings. (a) Bounded domains. In this setting, VS-APM displays linear convergence in inexact gradient steps, each of which requires utilizing an inner (SSG) scheme. Specifically, mVS-APM ach…
Classical theory for quasi-Newton schemes has focused on smooth deterministic unconstrained optimization while recent forays into stochastic convex optimization have largely resided in smooth, unconstrained, and strongly convex regimes. Naturally, there is a compelling need to address nonsmoothness, the lack of strong convexity, and the presence of constraints. Accordingly, this paper presents a quasi-Newton framework that can process merely convex and possibly nonsmooth (but smoothable) stochastic convex problems. We propose a framework that combines iterative smoothing and regularization wit…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Research, India（微软研究院印度分院）；University of Washington, Seattle（华盛顿大学西雅图分校）
We consider the classical setting of optimizing a nonsmooth Lipschitz continuous convex function over a convex constraint set, when having access to a (stochastic) first-order oracle (FO) for the function and a projection oracle (PO) for the constraint set. It is well known that to achieve $ε$-suboptimality in high-dimensions, $Θ(ε^{-2})$ FO calls are necessary. This is achieved by the projected subgradient method (PGD). However, PGD also entails $O(ε^{-2})$ PO calls, which may be computationally costlier than FO calls (e.g. nuclear norm constraints). Improving this PO calls complexity of PGD …
