Sign-based stochastic methods have gained attention due to their ability to achieve robust performance despite using only the sign information for parameter updates. However, the current convergence analysis of sign-based methods relies on the strong assumptions of first-order gradient Lipschitz and second-order gradient Lipschitz, which may not hold in practical tasks like deep neural network training that involve high non-smoothness. In this paper, we revisit sign-based methods and analyze their convergence under more realistic assumptions of first- and second-order smoothness. We first establish the convergence of the sign-based method under weak first-order Lipschitz. Motivated by the weak first-order Lipschitz, we propose a relaxed second-order condition that still allows for nonconvex acceleration in sign-based methods. Based on our theoretical results, we gain insights into the computational advantages of the recently developed LION algorithm. In distributed settings, we prove that this nonconvex acceleration persists with linear speedup in the number of nodes, when utilizing fast communication compression gossip protocols. The novelty of our theoretical results lies in that they are derived under much weaker assumptions, thereby expanding the provable applicability of sign-based algorithms to a wider range of problems.
MoSSP: A Momentum-Based Single-Loop Stochastic Penalty Method for Nonconvex Constrained DC-Regularized Optimization
School of Mathematical Sciences, Beihang University, Beijing 100191, China（北京航空航天大学数学学院）；School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China（中山大学计算机科学与工程学院）
Rethinking Cross-Modal Fine-Tuning: Optimizing the Interaction Between Feature Alignment and Target Fitting
Washington State University（华盛顿州立大学）；National University of Singapore（新加坡国立大学）；National Institute of Advanced Industrial Science and Technology（国家先进工业科学与技术研究院）；Hanoi University of Science and Technology（河内科学技术大学）
Descend or Rewind? Stochastic Gradient Descent Unlearning
Department of Engineering Sciences and Applied Mathematics, Northwestern University, Evanston, IL, United States（工程科学与应用数学系，西北大学，埃文斯顿，伊利诺伊州，美国）；Department of Industrial Engineering（工业工程系）；Applied Mathematics, Northwestern University, Evanston, IL, United States（应用数学，西北大学，埃文斯顿，伊利诺伊州，美国）
Breaking Correlation Shift via Conditional Invariant Regularizer
University of Chinese Academy of Sciences（中国科学院大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Langevin Multiplicative Weights Update with Applications in Polynomial Portfolio Management
School of Information Management & Engineering, Shanghai University of Finance and Economics（上海财经大学信息管理与工程学院）；College of Business, Shanghai University of Finance and Economics（上海财经大学商学院）
Global Convergence of Block Coordinate Descent in Deep Learning
Jiangxi Normal University（江西师范大学）；The Hong Kong University of Science and Technology（香港科技大学）；Northwestern University（西北大学）；City University of Hong Kong（香港城市大学）
Department of Computer Science & Engineering, University of California San Diego（计算机科学与工程系，加州大学圣地亚哥分校）；Department of Computer Sciences, University of Wisconsin-Madison（计算机科学系，威斯康星大学麦迪逊分校）；Archimedes, Athena Research Center, Greece（阿基米德，亚提卡研究中心，希腊）；Google DeepMind, London, UK（谷歌深Mind，伦敦，英国）
