We introduce LOREN, a curvature-aware zeroth-order (ZO) optimization method for fine-tuning large language models (LLMs). Existing ZO methods, which estimate gradients via finite differences using random perturbations, often suffer from high variance and suboptimal search directions. Our approach addresses these challenges by: (i) reformulating the problem of gradient preconditioning as that of adaptively estimating an anisotropic perturbation distribution for gradient estimation, (ii) capturing curvature through a low-rank block diagonal preconditioner using the framework of natural evolution strategies, and (iii) applying a REINFORCE leave-one-out (RLOO) gradient estimator to reduce variance. Experiments on standard LLM benchmarks show that our method outperforms state-of-the-art ZO methods by achieving higher accuracy and faster convergence, while cutting peak memory usage by up to 27.3% compared with MeZO-Adam.
Nanjing University of Information Science & Technology（南京信息工程大学）；Harbin Institute of Technology（哈尔滨工业大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
We consider escaping saddle points of nonconvex problems where only the function evaluations can be accessed. Although a variety of works have been proposed, the majority of them require either second or first-order information, and only a few of them have exploited zeroth-order methods, particularly the technique of negative curvature finding with zeroth-order methods which has been proven to be the most efficient method for escaping saddle points. To fill this gap, in this paper, we propose two zeroth-order negative curvature finding frameworks that can replace Hessian-vector product computa…
TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
Faculty of Engineering, The University of Sydney（悉尼大学工程学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络与空间安全学院）
Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures
Beijing International Center for Mathematical Research, Peking University（北京大学北京国际数学研究中心）；Center for Data Science, Peking University（北京大学数据科学中心）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）
CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China.（计算机科学与工程系，南方科技大学，深圳，中国）；Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.（数学系，南方科技大学，深圳，中国）
Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization
Michigan State University（密歇根州立大学）；University of Minnesota（明尼苏达大学）；University of California, Santa Barbara（加州大学圣芭芭拉分校）；DAMO Academy, Alibaba Group US（阿里巴巴集团美国达摩院）；IBM Research, USA（美国IBM研究院）
Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
Beijing Normal University（北京师范大学）；Singapore Management University（新加坡国立大学）；TAL Education Group（TAL教育集团）；Engineering Research Center of Intelligent Technology and Educational Application (MOE)（智能技术与教育应用工程研究中心（教育部））
Bosch Center for Artificial Intelligence（博世人工智能中心）；College of Information and Computer Sciences（信息与计算机科学学院）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Carnegie Mellon University（卡内基梅隆大学）
We study the problem of generating adversarial examples in a black-box setting, where we only have access to a zeroth order oracle, providing us with loss function evaluations. Although this setting has been investigated in previous work, most past approaches using zeroth order optimization implicitly assume that the gradients of the loss function with respect to the input images are \emph{unstructured}. In this work, we show that in fact substantial correlations exist within these gradients, and we propose to capture these correlations via a Gaussian Markov random field (GMRF). Given the intr…
University of Tübingen（蒂宾根大学）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）
Curvature in form of the Hessian or its generalized Gauss-Newton (GGN) approximation is valuable for algorithms that rely on a local model for the loss to train, compress, or explain deep networks. Existing methods based on implicit multiplication via automatic differentiation or Kronecker-factored block diagonal approximations do not consider noise in the mini-batch. We present ViViT, a curvature model that leverages the GGN's low-rank structure without further approximations. It allows for efficient computation of eigenvalues, eigenvectors, as well as per-sample first- and second-order direc…
CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models
Cornell University（康奈尔大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；Oak Ridge National Laboratory（橡树岭国家实验室）；Yale University（耶鲁大学）；University of California, Irvine（加州大学欧文分校）；Tulane University（路易斯安那大学塔兰帕伊斯分校）
Federated Learning on Riemannian Manifolds: A Gradient-Free Projection-Based Approach
School of Information Management and Engineering, Shanghai University of Finance and Economics（上海金融学院信息管理与工程学院）；School of Information Management and Engineering, Shanghai University of Finance and Economics（上海金融学院信息管理与工程学院；明尼苏达大学工业与系统工程系）；Department of Industrial and System Engineering, University of Minnesota（明尼苏达大学电气与计算机工程系）；Department of Electrical and Computer Engineering, University of Minnesota
University of California, Los Angeles（加州大学洛杉矶分校）；Colorado School of Mines（科罗拉多矿业学院）；University of Central Florida（中佛罗里达大学）；DAMO Academy, Alibaba US（阿里巴巴达摩院美国分部）
