Astra: 大语言模型激活空间尾特征值低秩适应
Astra: Activation-Space Tail-Eigenvector Low-Rank Adaptation of Large Language Models
AI总结:
Astra通过利用模型输出激活的尾特征向量构建低秩适配器,实现了更高效的参数高效微调,提升了下游任务性能。
摘要:
参数高效微调(PEFT)方法,尤其是LoRA,因其计算和存储效率而被广泛用于将预训练模型适应到下游任务中。然而,在LoRA及其变体的背景下,对应于尾特征向量的激活子空间的潜力仍然严重被低估,这可能导致微调性能不佳。在本工作中,我们提出了Astra(激活空间尾特征值低秩适应),一种新的PEFT方法,利用模型输出激活的尾特征向量(从一个小的任务特定校准集估计)来构建任务自适应的低秩适配器。通过将更新限制在由这些尾特征向量张成的子空间内,Astra在显著减少参数预算的情况下实现了更快的收敛速度和改进的下游性能。在自然语言理解(NLU)和自然语言生成(NLG)任务上的广泛实验表明,Astra在16个基准测试中一致优于现有的PEFT基线,并在某些情况下甚至超过了全微调(FFT)。\n
英文摘要:
Parameter-Efficient Fine-Tuning (PEFT) methods, especially LoRA, are widely used for adapting pre-trained models to downstream tasks due to their computational and storage efficiency. However, in the context of LoRA and its variants, the potential of activation subspaces corresponding to tail eigenvectors remains substantially under-exploited, which may lead to suboptimal fine-tuning performance. In this work, we propose Astra (Activation-Space Tail-Eigenvector Low-Rank Adaptation), a novel PEFT method that leverages the tail eigenvectors of the model output activations-estimated from a small task-specific calibration set-to construct task-adaptive low-rank adapters. By constraining updates to the subspace spanned by these tail eigenvectors, Astra achieves faster convergence and improved downstream performance with a significantly reduced parameter budget. Extensive experiments across natural language understanding (NLU) and natural language generation (NLG) tasks demonstrate that Astra consistently outperforms existing PEFT baselines across 16 benchmarks and even surpasses full fine-tuning (FFT) in certain scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。