arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.29317cs.CL
收藏

FoRA: 基于Fisher正交秩适配的参数高效微调

FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning

OptAI Inc(OptAI公司);LG Uplus

Juneyoung Park, Seongbae Lee, Han-Sang Lee, Kyuho Lee, Minjae Kim, Seungheon Hyeon, Kiduk Kwon, Seongwan Kim, Jaeho Lee

AI总结:

提出FoRA方法,通过Fisher信息选择信息层并在Stiefel流形上训练LoRA下投影,在减少参数预算的同时保持性能,优于LoRA和DoRA。

摘要:

参数高效微调(PEFT)主要关注LoRA及其面向精度的变体,而减少可训练参数的原始目标相对较少受到关注。我们引入了FoRA,通过减少适配层数而非适配器秩来重新审视这一目标。FoRA通过单次对角Fisher评分(训练成本低于1%)选择任务信息层,并在Stiefel流形上训练所选层的LoRA下投影,保持列正交性和有效秩。在五个LLaMA系列骨干网络上,FoRA在参数预算减半的情况下始终优于LoRA和DoRA,在参数数量为AdaLoRA四分之一时,精度差距在0.7-0.8个点以内。在来自LLaMA、Qwen3和Gemma系列的十二个骨干网络上的跨架构实验证实了从270M到32B参数的一致增益。两个组件超加性地结合:Fisher选择本身在相同预算下匹配秩缩减,而Stiefel约束提供了决定性的额外增益。

英文摘要:

Parameter-efficient fine-tuning(PEFT) has largely focused on LoRA and its accuracy-oriented variants, leaving the original goal of reducing trainable parameters has receivedcomparatively little attention. We introduce FoRA, which revisits this goal by reducing the number of adapted layers rather than adapter rank. FoRA selects task-informative layers via a single-pass diagonal Fisher score (under 1% of training cost) and trains the LoRA down-projection at selected layers on the Stiefel manifold, preserving column orthonormality and effective rank. FoRA consistently outperforms LoRA and DoRA at half their parameter budget, and falls within 0.7-0.8 accuracy points of AdaLoRA at one-quarter its parameter count, across five LLaMA-family backbones. Cross-architecture experiments on twelve backbones from the LLaMA, Qwen3, and Gemma families confirm consistent gains from 270M to 32B parameters. The two components combine super-additively: Fisher selection alone matches rank reduction at the same budget, while the Stiefel constraint provides the decisive additional gain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑