arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.20727cs.CL
收藏

ID-LoRA: 基于矩阵插值分解的高效低秩适应

ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition

Tianjin University(天津大学)

Xindian Ma, Rundong Kong, Peng Zhang, Ruoxiang Huang, Yongyu Jiang

AI总结:

ID-LoRA通过基于矩阵插值分解的低秩适应方法,在减少可训练参数的同时保持模型容量,优于现有PEFT技术。

摘要:

LoRA已成为一种通用的参数高效微调(PEFT)技术,使大型语言模型(LLMs)能够快速适应新任务。然而,当这些模型被放大时,即使是最新的LoRA变体仍然在可训练参数上引入了相当大的开销。相反,为了遏制这种开销而激进地降低秩会显著降低在复杂多任务设置中的性能。我们提出了ID-LoRA,一种新的PEFT框架,打破了这种权衡。其核心创新在于从预训练权重矩阵中提取并重用聚类参数组。这些组随后被用来形成多个低秩组件,所有这些组件仅共享一个初始化的可训练低秩矩阵。这种方法在减少可训练参数数量的同时保持了模型的容量。我们在五个多样的基准上评估了ID-LoRA:数学推理、代码生成、MMLU、常识问答和安全对齐。ID-LoRA在使用比标准LoRA少高达46%的可训练参数的情况下,优于完全微调和现有PEFT基线(例如LoRA、DoRA、HydraLoRA)。在多任务场景中,它在代码和MMLU任务上超过了LoRA及其最近的变体(例如DoRA和HydraLoRA),但仅需要传统LoRA所需的54%的可训练参数。

英文摘要:

LoRA has become a universal Parameter-Efficient Fine-Tuning (PEFT) technique that equips Large Language Models (LLMs) to adapt quickly to new tasks. However, when these models are scaled up, even the latest LoRA variants still introduce considerable overhead in trainable parameters. Conversely, aggressively lowering the rank to curb this overhead markedly degrades performance in complex multi-task settings. We propose ID-LoRA, a novel PEFT framework that breaks the trade-off. Its core innovation lies in extracting and reusing clustered parameter groups from the pretrained weight matrix. These groups are then used to form multiple low-rank components, all of which share only a single initialized trainable low-rank matrix. This approach cuts the number of trainable parameters while keeping the model's capacity intact. We evaluate ID-LoRA on five diverse benchmarks: Mathematical Reasoning, Code Generation, MMLU, CommonsenseQA, and Safety Alignment. ID-LoRA outperforms both full fine-tuning and existing PEFT baselines (e.g., LoRA, DoRA, HydraLoRA) while using up to 46% fewer trainable parameters than the standard LoRA. In multi-task scenarios, it surpasses LoRA and its recent variants (e.g., DoRA and HydraLoRA) on both Code and MMLU tasks, yet requires only 54% of the trainable parameters demanded by the conventional LoRA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑