PRoLoRA：部分旋转赋能更高参数效率的LoRA
PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA
AI总结：
本文提出PRoLoRA，一种通过广播缩减、旋转增强、部分共享细化和修正初始化实现层内共享的LoRA超集，在减少一半可训练参数的情况下仍优于LoRA，显著提升参数效率。
摘要：
随着大型语言模型（LLMs）的快速扩展，同时服务大量低秩适配（LoRA）已变得日益不切实际，导致成本难以承受，因此需要更高参数效率的微调方法。在本工作中，我们引入了部分旋转增强的低秩适配（PRoLoRA），这是一种层内共享机制，包含四个基本组件：广播缩减、旋转增强、部分共享细化以及修正初始化策略。作为LoRA的超集，PRoLoRA保留了其优势，并通过更优的模型容量、实际可行性和广泛适用性，有效规避了同类参数共享方法的缺点。实证实验表明，PRoLoRA在特定参数预算和性能目标场景下均展现出显著更高的参数效率，并且可扩展到更大的LLMs。值得注意的是，在可训练参数减少一倍的情况下，PRoLoRA在多个指令微调数据集上仍优于LoRA。随后，进行了消融研究，以验证各单独组件的必要性，并突出PRoLoRA相对于三种潜在变体的优越性。希望显著更高的参数效率能使PRoLoRA成为LoRA的资源友好型替代方案。
英文摘要：
With the rapid scaling of large language models (LLMs), serving numerous low-rank adaptations (LoRAs) concurrently has become increasingly impractical, leading to unaffordable costs and necessitating more parameter-efficient finetuning methods. In this work, we introduce Partially Rotation-enhanced Low-Rank Adaptation (PRoLoRA), an intra-layer sharing mechanism comprising four essential components: broadcast reduction, rotation enhancement, partially-sharing refinement, and rectified initialization strategy. As a superset of LoRA, PRoLoRA retains its advantages, and effectively circumvent the drawbacks of peer parameter-sharing methods with superior model capacity, practical feasibility, and broad applicability. Empirical experiments demonstrate the remarkably higher parameter efficiency of PRoLoRA in both specific parameter budget and performance target scenarios, and its scalability to larger LLMs. Notably, with one time less trainable parameters, PRoLoRA still outperforms LoRA on multiple instruction tuning datasets. Subsequently, an ablation study is conducted to validate the necessity of individual components and highlight the superiority of PRoLoRA over three potential variants. Hopefully, the conspicuously higher parameter efficiency can establish PRoLoRA as a resource-friendly alternative to LoRA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
