This paper is devoted to studying the first-order variational analysis of non-convex and non-differentiable functions that may not be subdifferentially regular. To achieve this goal, we entirely rely on two concepts of directional derivatives known as subderivative and semi-derivative. We establish the exact chain and sum rules for this class of functions via these directional derivatives. These calculus rules provide an implementable auto-differentiation process such as back-propagation in composite functions. The latter calculus rules can be used to identify the directional stationary points defined by the subderivative. We show that the distance function of a geometrically derivable constraint set is semi-differentiable, which opens the door for designing first-order algorithms for non-Clarke regular constrained optimization problems. We propose a first-order algorithm to find a directional stationary point of non-Clarke regular and perhaps non-Lipschitz functions. We introduce a descent property under which we establish the non-asymptotic convergence of our method with rate $O(\varepsilon^{-2})$, akin to gradient descent for smooth minimization. We show that the latter descent property holds for free in some interesting non-amenable composite functions, in particular, it holds for the Moreau envelope of any bounded-below function.
High-Probability Convergence for Composite and Distributed Stochastic Minimization and Variational Inequalities with Heavy-Tailed Noise
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Université de Montréal（蒙特利尔大学）；Mila（米拉研究所）；Weierstrass Institute for Applied Analysis and Stochastics（魏尔斯特拉斯应用分析与随机过程研究所）；Ivannikov Institute for System Programming RAS（伊万尼科夫系统编程研究所（俄罗斯科学院））；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）
