MASA:重新思考LoRA中的表示瓶颈问题:多A共享适应
MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptation
AI总结:
MASA通过多A共享适应架构提升LoRA在多领域泛化和多任务推理中的表现,实现更高准确率
摘要:
低秩适应(LoRA)已成为参数高效微调(PEFT)中大型语言模型的主要方法,它通过在transformer层中添加一个下投影矩阵A和一个上投影矩阵B来增强。然而,LoRA依赖于单一的下投影矩阵(A)会创造出表示瓶颈,因为这个单一的特征提取器本质上不足以捕捉复杂任务所需的多样化信号。这促使我们进行架构转变,以关注丰富特征适应来提高下游任务适应能力。我们提出了MASA(多A共享适应),一种实现多A、单B结构的架构,其中多A专家集合在层之间非对称共享以确保参数效率。在MASA中,这些专门的专家捕捉多样化的特征,然后由一个单层特定的B矩阵进行整合。我们方法的有效性和通用性通过涵盖多领域泛化、单领域专业化和多任务推理的全面实验集得到验证。例如,在MMLU基准测试中,MASA实现了平均准确率59.62%,在可比的可学习参数0.52下,比标准LoRA高出1.08点(相对提高1.84%)
英文摘要:
Low-Rank Adaptation (LoRA) has emerged as a dominant method in Parameter-Efficient Fine-Tuning (PEFT) for large language models, which augments the transformer layer with one down-projection $A$ and one up-projection $B$. However, LoRA's reliance on a single down-projection matrix ($A$) creates a representational bottleneck, as this solitary feature extractor is inherently insufficient for capturing the diverse signals required by complex tasks. This motivates our architectural shift to focus on enriching the feature adaptation to improve the downstream task adaptation ability. We propose MASA (Multi-$A$ Shared Adaptation), an architecture that implements a multi-$A$, single-$B$ structure where the multi-$A$ expert ensemble is asymmetrically shared across layers to ensure parameter efficiency. In MASA, these specialized experts capture diverse features, which are then integrated by a single, layer-specific $B$-matrix. The effectiveness and versatility of our method are validated through a comprehensive suite of experiments spanning multi-domain generalization, single-domain specialization, and multi-task reasoning. For example, on the MMLU benchmark, MASA achieves an average accuracy of 59.62%, outperforming the standard LoRA by 1.08 points (a relative improvement of 1.84%) with comparable learnable parameters of 0.52%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。