arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.11239cs.LG
收藏

从低秩梯度子空间稳定化到低秩权重:观察、理论与应用

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications

University of Texas at Austin(德克萨斯大学奥斯汀分校);Eindhoven University of Technology(埃因霍温理工大学);University of Surrey(萨里大学);University of Oxford(牛津大学);California Institute of Technology(加州理工学院)

Ajay Jaiswal, Yifan Wang, Lu Yin, Shiwei Liu, Runjin Chen, Jiawei Zhao, Ananth Grama, Yuandong Tian, Zhangyang Wang

AI总结:

本文从梯度子空间稳定化揭示 LLM 权重的非均匀低秩特性,并提出 WeLore,实现数据无关的一次性权重压缩与内存高效微调。

摘要:

大型语言模型(LLM)的权重矩阵通常可以表示为低秩形式,从而有望缓解内存和计算资源需求。不同于以往专注于开发新型矩阵分解的工作,本文通过稳定梯度子空间的视角研究 LLM 中权重矩阵的非均匀低秩特性。首先,我们提供了一个理论框架,通过 Hessian 分析理解梯度子空间的稳定化。其次,我们通过实验确立了梯度动态与权重矩阵低秩表达能力之间的重要关系。研究发现表明,不同 LLM 组件呈现出不同程度的收敛低秩结构,因此需要在它们之间采用可变秩缩减,以尽量降低压缩导致的性能下降。基于该结果,我们提出了 Weight Low-Rank Projection(WeLore),它以数据无关且一次性的方式将权重压缩与内存高效微调统一起来。当作为压缩技术使用时,WeLore 将权重矩阵分为低秩组件(LRC)和非低秩组件(N-LRC),并对其进行适当编码,以实现最小的性能损失。我们的梯度动态视角表明,LRC 往往具有更好的微调能力,对其单独微调可以紧密逼近、有时甚至超过全量微调的训练损失轨迹和性能,同时显著减少内存与计算占用。代码可在 https://github.com/VITA-Group/WeLore 获取。

英文摘要:

Large Language Models' (LLMs) weight matrices can often be expressed in low-rank form with potential to relax memory and compute resource requirements. Unlike prior efforts that focus on developing novel matrix decompositions, in this work we study the non-uniform low-rank properties of weight matrices in LLMs through the lens of stabilizing gradient subspace. First, we provide a theoretical framework to understand the stabilization of gradient subspaces through Hessian analysis. Second, we empirically establish an important relationship between gradient dynamics and low-rank expressiveness of weight matrices. Our findings reveal that different LLM components exhibit varying levels of converged low-rank structures, necessitating variable rank reduction across them to minimize drop in performance due to compression. Drawing on this result, we present Weight Low-Rank Projection(WeLore) that unifies weight compression and memory-efficient fine-tuning into one, in a data-agnostic and one-shot manner. When used as a compression technique, WeLore categorizes weight matrices into Low-rank Components (LRCs) and Non-Low-rank Components (N-LRCs) and suitably encodes them for minimum performance loss. Our gradient dynamics perspective illustrates that LRCs tend to have better fine-tuning capabilities and their standalone fine-tuning can closely mimic and sometimes outperform the training loss trajectory and performance of full fine-tuning with notable memory and compute footprint reduction. Codes are available at https://github.com/VITA-Group/WeLore.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑