A Geometry-Aware Efficient Algorithm for Compositional Entropic Risk Minimization
Texas A\&M University（德克萨斯A&M大学）；University of Texas, Austin（德克萨斯大学奥斯汀分校）；National Taiwan University（国立台湾大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Xiyuan Wei, Linli Zhou, Bokun Wang, Chih-Jen Lin, Tianbao Yang
This paper studies optimization for a family of problems termed $\textbf{compositional entropic risk minimization}$, in which each data's loss is formulated as a Log-Expectation-Exponential (Log-E-Exp) function. The Log-E-Exp formulation serves as an abstraction of the Log-Sum-Exponential (LogSumExp) function when the explicit summation inside the logarithm is taken over a gigantic number of items and is therefore expensive to evaluate. While entropic risk objectives of this form arise in many machine learning problems, existing optimization algorithms suffer from several fundamental limitations including non-convergence, numerical instability, and slow convergence rates. To address these limitations, we propose a geometry-aware stochastic algorithm, termed $\textbf{SCENT}$, for the dual formulation of entropic risk minimization cast as a min--min optimization problem. The key to our design is a $\textbf{stochastic proximal mirror descent (SPMD)}$ update for the dual variable, equipped with a Bregman divergence induced by a negative exponential function that faithfully captures the geometry of the objective. Our main contributions are threefold: (i) we establish an $O(1/\sqrt{T})$ convergence rate of the proposed SCENT algorithm for convex problems; (ii) we theoretically characterize the advantages of SPMD over standard SGD update for optimizing the dual variable; and (iii) we demonstrate the empirical effectiveness of SCENT on extreme classification, partial AUC maximization, contrastive learning and distributionally robust optimization, where it consistently outperforms existing baselines. Code is available at https://github.com/Optimization-AI/SCENT.
Compositional Stochastic Average Gradient for Machine Learning and Related Applications
The Pennsylvania State University（宾夕法尼亚州立大学）；College of Information Science and Technology（信息科学与技术学院）；Department of Computer Science and Engineering（计算机科学与工程系）
Provable Responsible AI and Data Analytics (PRADA) Lab（可证明责任AI和数据分析实验室）；King Abdullah University of Science and Technology（卡瓦尔大学科学与技术大学）；State University of New York at Buffalo（纽约州立大学布法罗分校）；Technische Universität Berlin（柏林技术大学）
