arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.03230cs.LGcs.AIcs.CLmath.OC
收藏

DiaBlo: 对角块足以用于微调

DiaBlo: Diagonal Blocks Are Sufficient For Finetuning

University at Albany, SUNY(纽约州立大学阿尔巴尼分校);IBM T. J. Watson Research Center(IBM 汤普逊·杰·沃森研究中心)

Selcuk Gurses, Aozhong Zhang, Yanxia Deng, Xun Dong, Xin Li, Naigang Wang, Penghang Yin, Zi Yang

AI总结:

DiaBlo是一种仅更新模型权重矩阵对角块的参数高效微调方法,通过消除低秩矩阵乘积需求,实现稳定收敛和高效训练。

摘要:

微调是将大型语言模型(LLMs)适应特定领域下游任务的关键步骤。为了减轻全模型微调的显著计算和内存成本,参数高效微调(PEFT)方法被提出,以仅更新一小部分模型参数。然而,PEFT方法与全模型微调之间仍存在性能差距。在本工作中,我们提出了DiaBlo,一种简单而有效的PEFT方法,仅更新所选模型权重矩阵的对角块。与低秩适应(LoRA)及其变体不同,DiaBlo消除了对低秩矩阵乘积的需要,从而避免了依赖辅助初始化方案或定制优化策略来提高收敛性。这种设计导致了稳定且稳健的收敛,同时保持了与LoRA相当的内存效率和训练速度。此外,我们提供了理论保证,表明在温和的低秩条件下,DiaBlo在线性问题中比LoRA更具表现力,并且收敛到一般非线性全微调的 stationary 点。通过在一系列任务上的广泛实验,包括常识推理、算术推理、代码生成和安全对齐,我们展示了仅微调对角块足以实现强大且一致的性能。DiaBlo不仅实现了竞争性的准确性,还保持了高内存效率和快速的微调速度。代码可在 https://github.com/ziyangjoy/DiaBlo 上获得。

英文摘要:

Fine-tuning is a critical step for adapting large language models (LLMs) to domain-specific downstream tasks. To mitigate the substantial computational and memory costs of full-model fine-tuning, Parameter-Efficient Fine-Tuning (PEFT) methods have been proposed to update only a small subset of model parameters. However, performance gaps between PEFT approaches and full-model fine-tuning still exist. In this work, we present DiaBlo, a simple yet effective PEFT approach that updates only the diagonal blocks of selected model weight matrices. Unlike Low-Rank Adaptation (LoRA) and its variants, DiaBlo eliminates the need for low-rank matrix products, thereby avoiding the reliance on auxiliary initialization schemes or customized optimization strategies to improve convergence. This design leads to stable and robust convergence while maintaining comparable memory efficiency and training speed to LoRA. Moreover, we provide theoretical guarantees showing that, under mild low-rank conditions, DiaBlo is more expressive than LoRA in the linear problem and converges to a stationary point of the general nonlinear full fine-tuning. Through extensive experiments across a range of tasks, including commonsense reasoning, arithmetic reasoning, code generation, and safety alignment, we show that fine-tuning only diagonal blocks is sufficient for strong and consistent performance. DiaBlo not only achieves competitive accuracy but also preserves high memory efficiency and fast fine-tuning speed. Codes are available at https://github.com/ziyangjoy/DiaBlo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑