arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13659cs.LGmath.OC
收藏

通过可学习方向采样实现LLM微调的零阶优化

Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling

Valery Parfenov, Grigoriy Evseev, Andrey Veprikov, Nikolay Bushkov, Stanislav Moiseev, Aleksandr Beznosikov

AI总结:

本文提出了一种基于可学习策略的方向采样方法,用于改进零阶优化在LLM微调中的性能,通过减少方差和依赖维度来提升效果。

摘要:

对大型预训练语言模型(LLMs)进行微调是现代NLP的基石,但其快速增长的内存需求(由反向传播和大型优化器状态驱动)限制了在资源受限环境中的部署。零阶(ZO)方法通过从正向评估中估计方向导数来绕过反向传播,提供了显著的内存节省。然而,传统ZO估计器存在方差高且对参数维度性$d$有不良依赖,这限制了其在低维问题中的应用。在本工作中,我们提出了一种策略驱动的ZO框架,将扰动方向上的采样分布视为可学习的策略,并更新它以减少方向估计的方差。我们开发了一个实用的算法实现这一想法,并提供理论分析,显示学习的采样分布提高了梯度信息的质量并放松了收敛界限中对$d$的显式依赖。实验上,我们在具有挑战性的LLM微调基准上验证了该方法,证明其比标准ZO基线有显著改进的性能。我们的结果表明,自适应方向采样是使大规模ZO微调可行的有前途的途径。源代码可在https://github.com/brain-lab-research/zo_ldsd获取。

英文摘要:

Fine-tuning large pretrained language models (LLMs) is a cornerstone of modern NLP, yet its growing memory demands (driven by backpropagation and large optimizer States) limit deployment in resource-constrained settings. Zero-order (ZO) methods bypass backpropagation by estimating directional derivatives from forward evaluations, offering substantial memory savings. However, classical ZO estimators suffer from high variance and an adverse dependence on the parameter dimensionality $d$, which has constrained their use to low-dimensional problems. In this work, we propose a policy-driven ZO framework that treats the sampling distribution over perturbation directions as a learnable policy and updates it to reduce the variance of directional estimates. We develop a practical algorithm implementing this idea and provide a theoretical analysis, showing that learned sampling distributions improve the quality of gradient information and relax the explicit dependence on $d$ in convergence bounds. Empirically, we validate the approach on challenging LLM fine-tuning benchmarks, demonstrating substantially improved performance compared to standard ZO baselines. Our results suggest that adaptive direction sampling is a promising route to make ZO fine-tuning viable at scale. The source code is available at https://github.com/brain-lab-research/zo_ldsd

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑