arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.01419cs.CLcs.AI
收藏

通过子空间中的反卷积实现大型语言模型的参数高效微调

Parameter-Efficient Fine-Tuning of Large Language Models via Deconvolution in Subspace

School of Electronic and Electrical Engineering, Shanghai University of Engineering Science(上海工程技术大学电子与电气工程学院)

Jia-Chen Zhang, Yu-Jie Xiong, Chun-Ming Xia, Dong-Hai Zhu, Xi-He Qiu

AI总结:

本文提出了一种通过子空间反卷积进行参数高效微调的新方法DCFT,利用反卷积补全细节并增强子空间增量矩阵知识,通过调整核大小动态控制参数,突破了LoRA秩一分解的瓶颈,在实现8倍参数减少的同时保持优异性能。

摘要:

大型语言模型(LLM)被认为是迈向通用人工智能(AGI)的里程碑。凭借其先进的涌现能力,它能适应广泛的具体应用。针对各种下游任务微调LLM已成为一种新范式。低秩适应(LoRA)以其参数高效性而闻名,它可以将微调LLM所需的参数量减少几个数量级。然而,基于LoRA的方法由于秩一分解造成的瓶颈而面临重大局限。随着LLM中参数量的增加,即使是秩一分解也可能超过处理更多下游任务真正所需的参数数量。在本文中,我们提出了一种通过子空间反卷积进行参数高效微调(PEFT)的新方法,称为DCFT。我们创新性地使用反卷积来补全细节并增强子空间增量矩阵中的知识,并通过调整核大小动态控制参数,不受秩一分解的约束。我们进行了大量实验以验证DCFT的有效性。结果表明,与LoRA相比,DCFT实现了8倍的参数减少,且仍取得了令人瞩目的性能。

英文摘要:

Large language model (LLM) is considered a milestone towards achieving Artificial General Intelligence (AGI). With its advanced emergent capabilities, it adapt to a wide range of specific applications. Fine-tuning LLMs for various downstream tasks has become a new paradigm. Low-Rank Adaptation (LoRA) is well-known for its parameter efficiency. It can reduce the number of parameters needed to fine-tune LLMs by several orders of magnitude. However, LoRA-based approaches encounter a significant limitation due to the bottleneck imposed by rank one decomposition. As the parameters count in LLMs increase, even rank one decomposition might surpass the number of parameters truly necessary for handling more downstream tasks. In this paper, we propose a new method for Parameter-Efficient Fine-Tuning (PEFT) via deconvolution in subspace, dubbed as DCFT. We innovatively use deconvolution to complete details and enhance knowledge in subspace incremental matrices, and dynamically control parameters by adjusting the kernel size, unconstrained by rank-one decomposition. Extensive experiments are conducted to validate the effectiveness of DCFT. Results show that compared to LoRA, DCFT achieve an 8$\times$ reduction in parameters, and still achieves highly impressive performance. Our code is available here: https://github.com/Godz-z/DCFT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑