Gradient-based optimization methods are commonly used to identify local optima in high-dimensional spaces. When derivatives cannot be evaluated directly, stochastic estimators can provide approximate gradients. However, these estimators' perturbation-based sampling of the objective function introduces variance that can lead to slow convergence. In this paper, we present dimensional peeking, a variance reduction method for gradient estimation in discrete optimization via simulation. By lifting the sampling granularity from scalar values to classes of values that follow the same control flow path, we increase the information gathered per simulation evaluation. Our derivation from an established smoothed gradient estimator shows that the method does not introduce any bias. We present an implementation via a custom numerical data type to transparently carry out dimensional peeking over C++ programs. Variance reductions by factors of up to 7.9 are observed for three simulation-based optimization problems with high-dimensional input. The optimization progress compared to three meta-heuristics shows that dimensional peeking increases the competitiveness of zeroth-order optimization for discrete and non-convex simulations.
An Accelerated Variance-Reduced Conditional Gradient Sliding Algorithm for First-order and Zeroth-order Optimization
Nanjing University of Information Science & Technology(南京信息工程大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);JD Finance America Corporation(京东金融美国公司);University of Pittsburgh(匹兹堡大学)
Zeroth-Order Non-Log-Concave Sampling with Variance Reduction and Applications to Inverse Problems
Elmore School of Electrical and Computer Engineering, Purdue University, West Lafayette, USA(电子工程学院,普渡大学,西拉法叶,美国);Weldon School of Biomedical Engineering, Purdue University, West Lafayette, USA(生物医学工程学院,普渡大学,西拉法叶,美国)
Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization
MIT-IBM Watson AI Lab(麻省理工学院-IBM Watson人工智能实验室);IBM Research(IBM研究);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);University of Michigan, Ann Arbor(密歇根大学安娜堡分校)
Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization
The Ohio State University(俄亥俄州立大学);Duke University(杜克大学)
基于一阶技术SVRG和SARAH/SPIDER,最近分别设计了两类求解非凸优化问题的零阶随机算法。本文讨论了几个重要的问题,这些方法仍然是开放的。首先,所有现有的SVRG型零阶算法遭受更差的函数查询复杂度比零阶梯度下降(ZO-GD)或随机梯度下降(ZO-SGD)。在本文中,我们提出了一种新算法ZO-SVRG-Coord-Rand,并对Liu et al. 2018 b中提出的现有ZO-SVRG-Coord算法进行了新的分析,并表明ZO-SVRG-Coord-Rand和ZO-SVRG-Coord(在我们的新分析下)优于其他现有的SVRG型零阶方法以及ZO-GD和ZO-SGD。其次,现有的SPIDER型算法SPIDER-SZO(Fang et al. 2018)具有优越的理论性能,但在实践中会产生大量高斯随机变量以及$\sqrtε$级步长。在本文中,我们开发了一个新的算法ZO-SPIDER-Coord,这是免费的高斯变量的生成,并允许一个大的常数步长,同时保持相同的收敛速度和查询复杂度,我们进一步表明,ZO-SPIDER-Coord自动实现线性收敛速度,因为搜索器进入一个本地PL区域没有重新启动和算法修改。
Zeroth-Order Methods for Nonconvex Stochastic Problems with Decision-Dependent Distributions
Graduate School of Information Science and Technology, University of Tokyo(东京大学信息科学与技术研究生院);Center for Advanced Intelligence Project, RIKEN(理化学研究所先进智能项目中心)
Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNs
Tianjin University National Center for Applied Mathematics(天津大学应用数学中心);Jilin University School of Mechanical and Aerospace Engineering(吉林大学机械与 aerospace 工程学院)
Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradient
School of Management, Xi’an Jiaotong University(西安交通大学管理学院);State Grid Corporation of China(中国国家电网有限公司);International Business School, Beijing Foreign Studies University(北京外国语大学国际商学院);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);College of Computing and Data Science, NTU(南洋理工大学计算机与数据科学学院)
Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
Beijing Normal University(北京师范大学);Singapore Management University(新加坡国立大学);TAL Education Group(TAL教育集团);Engineering Research Center of Intelligent Technology and Educational Application (MOE)(智能技术与教育应用工程研究中心(教育部))
CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China.(计算机科学与工程系,南方科技大学,深圳,中国);Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.(数学系,南方科技大学,深圳,中国)
Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling
South China University of Technology(华南理工大学);JD Explore Academy(京东探索研究院);Shenzhen Campus of Sun Yat-sen University(深圳校区中山大学);University of Sydney(悉尼大学);Center for AI Theoretical Foundation and Systems, Shenzhen Loop Area Institute(人工智能理论基础与系统中心,深圳loop区研究院)
Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent
Chongqing University(重庆大学);North Carolina State University(北卡罗来纳州立大学);Oklahoma State University(俄克拉荷马州立大学);DEVCOM Army Research Laboratory(DEVCOM陆军研究实验室)
Zeroth-Order Alternating Gradient Descent Ascent Algorithms for a Class of Nonconvex-Nonconcave Minimax Problems
Shanghai University(上海大学);Newtouch Center for Mathematics of Shanghai University(上海大学纽崔克数学中心);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院)
How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspective
Michigan State University(密歇根州立大学);JD AI Research(京东人工智能研究院);University of Minnesota(明尼苏达大学);UC Santa Barbara(加州大学圣巴巴拉分校);MIT-IBM Watson AI Lab, IBM Research(IBM研究中心MIT-IBM沃森人工智能实验室)
Stochastic Zeroth-Order Optimization under Strongly Convexity and Lipschitz Hessian: Minimax Sample Complexity
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);University of Texas at Dallas(德克萨斯大学达拉斯分校);University of California, Berkeley(加利福尼亚大学伯克利分校);New York University(纽约大学);Princeton University(普林斯顿大学)
Minimisation of Quasar-Convex Functions Using Random Zeroth-Order Oracles
School of Engineering(工程学院);Australian National University(澳大利亚国立大学);Department of Electrical and Electronic Engineering(电子与电气工程系);University of Melbourne(墨尔本大学)
Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks
Peking University(北京大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Peng Cheng Laboratory(鹏城实验室)
Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization
Michigan State University(密歇根州立大学);University of Minnesota(明尼苏达大学);University of California, Santa Barbara(加州大学圣芭芭拉分校);DAMO Academy, Alibaba Group US(阿里巴巴集团美国达摩院);IBM Research, USA(美国IBM研究院)
Accelerated Stochastic Gradient-free and Projection-free Methods
College of Computer Science & Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院);MIIT Key Laboratory of Pattern Analysis & Machine Intelligence(工信部模式分析与机器智能重点实验室)
Hi-ZFO: Hierarchical Zeroth- and First-Order LLM Fine-Tuning via Importance-Guided Tensor Selection
School of Intelligence Science and Technology, Peking University(智能科学与技术学院,北京大学);State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室)
The Multi-Query Paradox in Zeroth-Order Optimization
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong(香港中文大学计算机科学与工程系);The Chinese University of Hong Kong, Hong Kong(香港中文大学);Xiamen University, China(厦门大学)
Communication-Efficient and Differentially Private Vertical Federated Learning with Zeroth-Order Optimization
Elmore Family School of Electrical and Computer Engineering, Purdue University(电气与计算机工程学院,普渡大学);Department of Computer Science and Engineering, Yonsei University(计算机科学与工程系,延世大学)
Obtaining Lower Query Complexities through Lightweight Zeroth-Order Proximal Gradient Algorithms
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Texas A&M University(得克萨斯农工大学);University Maryland College Park(马里兰大学学院公园分校)
Zeroth-Order Actor-Critic: An Evolutionary Framework for Sequential Decision Problems
The University of Hong Kong(香港大学);Tsinghua University(清华大学);SunRising AI Ltd.(森瑞昇人工智能有限公司);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Shanghai Qi Zhi Institute(上海期智研究院)