While fine-tuning large language models (LLMs) for specific tasks often yields impressive results, it comes at the cost of memory inefficiency due to back-propagation in gradient-based training. Memory-efficient Zeroth-order (MeZO) optimizers, recently proposed to address this issue, only require forward passes during training, making them more memory-friendly. However, compared with exact gradients, ZO-based gradients usually exhibit an estimation error, which can significantly hurt the optimization process, leading to slower convergence and suboptimal solutions. In addition, we find that the estimation error will hurt more when adding to large weights instead of small weights. Based on this observation, this paper introduces Sparse MeZO, a novel memory-efficient zeroth-order optimization approach that applies ZO only to a carefully chosen subset of parameters. We propose a simple yet effective parameter selection scheme that yields significant performance gains with Sparse-MeZO. Additionally, we develop a memory-optimized implementation for sparse masking, ensuring the algorithm requires only inference-level memory consumption, allowing Sparse-MeZO to fine-tune LLaMA-30b on a single A100 GPU. Experimental results illustrate that Sparse-MeZO consistently improves both performance and convergence speed over MeZO without any overhead. For example, it achieves a 9\% absolute accuracy improvement and 3.5x speedup over MeZO on the RTE task. Code is available at https://github.com/NUS-HPC-AI-Lab/SparseMeZO.
Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
Beijing Normal University（北京师范大学）；Singapore Management University（新加坡国立大学）；TAL Education Group（TAL教育集团）；Engineering Research Center of Intelligent Technology and Educational Application (MOE)（智能技术与教育应用工程研究中心（教育部））
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
Michigan State University（密歇根州立大学）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；UT Austin（德克萨斯大学奥斯汀分校）；University of Minnesota Twin Cities（明尼苏达大学双城分校）；IBM Research（IBM研究院）；Princeton University（普林斯顿大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；MIT（麻省理工学院）；Harvard University（哈佛大学）
CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China.（计算机科学与工程系，南方科技大学，深圳，中国）；Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.（数学系，南方科技大学，深圳，中国）
On-Device Fine-Tuning via Backprop-Free Zeroth-Order Optimization
Institute for Intelligent Networked Systems, Northeastern University London（智能网络系统研究所，东北大学伦敦分校）；Department of Engineering, King’s College London（工程系，伦敦大学国王学院）
Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning
School of Intelligence Science and Technology（智能科学与技术学院）；Institute for Artificial Intelligence（人工智能研究院）；Peking University（北京大学）；State Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）
TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
Faculty of Engineering, The University of Sydney（悉尼大学工程学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络与空间安全学院）
Stochastic zeroth-order (SZO), or gradient-free, optimization allows to optimize arbitrary functions by relying only on function evaluations under parameter perturbations, however, the iteration complexity of SZO methods suffers a factor proportional to the dimensionality of the perturbed function. We show that in scenarios with natural sparsity patterns as in structured prediction applications, this factor can be reduced to the expected number of active features over input-output pairs. We give a general proof that applies sparse SZO optimization to Lipschitz-continuous, nonconvex, stochastic…
Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization
Michigan State University（密歇根州立大学）；University of Minnesota（明尼苏达大学）；University of California, Santa Barbara（加州大学圣芭芭拉分校）；DAMO Academy, Alibaba Group US（阿里巴巴集团美国达摩院）；IBM Research, USA（美国IBM研究院）
AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning
Southern University of Science and Technology（南方科技大学）；National University of Singapore（新加坡国立大学）；Huawei Technologies Ltd.（华为技术有限公司）；City University of Hong Kong（香港城市大学）
Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；University of Houston（休斯顿大学）；State University of New York at Binghamton（纽约州立大学布法罗分校）；Arizona State University（亚利桑那州立大学）；Department of Artificial Intelligence and Informatics, Mayo Clinic（梅奥诊所人工智能与信息学系）
Interest in stochastic zeroth-order (SZO) methods has recently been revived in black-box optimization scenarios such as adversarial black-box attacks to deep neural networks. SZO methods only require the ability to evaluate the objective function at random input points, however, their weakness is the dependency of their convergence speed on the dimensionality of the function to be evaluated. We present a sparse SZO optimization method that reduces this factor to the expected dimensionality of the random perturbation during learning. We give a proof that justifies this reduction for sparse SZO …
University of California, Los Angeles（加州大学洛杉矶分校）；Damo Academy, Alibaba US（阿里巴巴美国达摩院）；Xmotors AI（Xmotors人工智能公司）
We consider the problem of minimizing a high-dimensional objective function, which may include a regularization term, using (possibly noisy) evaluations of the function. Such optimization is also called derivative-free, zeroth-order, or black-box optimization. We propose a new $\textbf{Z}$eroth-$\textbf{O}$rder $\textbf{R}$egularized $\textbf{O}$ptimization method, dubbed ZORO. When the underlying gradient is approximately sparse at an iterate, ZORO needs very few objective function evaluations to obtain a new iterate that decreases the objective function. We achieve this with an adaptive, ran…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Synopsys, Inc.（新思科技）；Alpine Optoelectronics, Inc.（Alpine光电子公司）
Optical neural networks (ONNs) have demonstrated record-breaking potential in high-performance neuromorphic computing due to their ultra-high execution speed and low energy consumption. However, current learning protocols fail to provide scalable and efficient solutions to photonic circuit optimization in practical applications. In this work, we propose a novel on-chip learning framework to release the full potential of ONNs for power-efficient in situ training. Instead of deploying implementation-costly back-propagation, we directly optimize the device configurations with computation budgets …
Zeroth-Order Methods for Stochastic Nonconvex Nonsmooth Composite Optimization
Department of Computer Science University of Maryland College Park（计算机科学系大学马里兰大学学院公园分校）；Department of Computer Science and Engineering University of Texas Arlington（计算机科学与工程系德克萨斯大学阿灵顿分校）
This paper considers stochastic convex optimization problems with two sets of constraints: (a) deterministic constraints on the domain of the optimization variable, which are difficult to project onto; and (b) deterministic or stochastic constraints that admit efficient projection. Problems of this form arise frequently in the context of semidefinite programming as well as when various NP-hard problems are solved approximately via semidefinite relaxation. Since projection onto the first set of constraints is difficult, it becomes necessary to explore projection-free algorithms, such as the sto…
Modern-day neural networks are famously large, yet also highly redundant and compressible; there exist numerous pruning strategies in the deep learning literature that yield over 90% sparser sub-networks of fully-trained, dense architectures while still maintaining their original accuracies. Amongst these many methods though -- thanks to its conceptual simplicity, ease of implementation, and efficacy -- Iterative Magnitude Pruning (IMP) dominates in practice and is the de facto baseline to beat in the pruning community. However, theoretical explanations as to why a simplistic method such as IM…
ZO-DARTS++: An Efficient and Size-Variable Zeroth-Order Neural Architecture Search Algorithm
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB), Politecnico di Milano（米兰理工大学电子、信息与生物工程系）
Derivative-free Alternating Projection Algorithms for General Nonconvex-Concave Minimax Problems
Shanghai University（上海大学）；Newtouch Center for Mathematics of Shanghai University（上海大学新触数学中心）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
Shanghai Key Laboratory of Intelligent Information Processing（上海智能信息处理重点实验室）；School of Computer Science, Fudan University（复旦大学计算机科学学院）
We explore the black-box adversarial attack on video recognition models. Attacks are only performed on selected key regions and key frames to reduce the high computation cost of searching adversarial perturbations on a video due to its high dimensionality. To select key frames, one way is to use heuristic algorithms to evaluate the importance of each frame and choose the essential ones. However, it is time inefficient on sorting and searching. In order to speed up the attack process, we propose a reinforcement learning based frame selection strategy. Specifically, the agent explores the differ…
