BA-LoRA: 降低偏差的低秩适应以减轻大语言模型中的灾难性继承
BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models
AI总结：
本文提出BA-LoRA，通过引入一致性、多样性及SVD项正则化，缓解低秩适应中的灾难性继承问题，提升模型鲁棒性和公平性。
摘要：
参数高效微调（PEFT）已成为调整大语言模型（LLMs）的标准方法。然而，我们发现流行低秩适应方法如LoRA存在关键漏洞：它们会加剧“灾难性继承”——即从预训练中不可控传播的偏见、噪声和数据不平衡。这种现象会降低模型的鲁棒性和公平性，削弱高效适应的好处。为此，我们引入了偏见缓解低秩适应（BA-LoRA）。我们的方法基于对灾难性继承的系统分解，将其归因于三个核心挑战：知识漂移、表征崩溃和对噪声的过拟合。BA-LoRA通过引入一致性、多样性和基于SVD的三项目标正则化来系统缓解这些问题：保持核心知识、促进表征丰富性以及鼓励稳健的低秩输出表示。我们对一系列自然语言生成（NLG）和自然语言理解（NLU）任务进行了全面评估，使用多样且知名的开源语言模型（如LLaMA-2-7B和DeBERTa-v3-base）。我们的结果表明，BA-LoRA不仅在性能和稳定性上优于最先进的LoRA变体，而且在针对评估中表现出更强的鲁棒性和偏见缓解能力。这些结果提供了证据，表明BA-LoRA可以抵消灾难性继承的负面影响。
英文摘要：
Parameter-efficient fine-tuning (PEFT) has become a de facto standard for adapting large language models (LLMs). However, we identify a critical vulnerability within popular low-rank adaptation methods such as LoRA: they can exacerbate "Catastrophic Inheritance" - the unchecked propagation of biases, noise, and data imbalances from pre-training. This phenomenon can degrade model robustness and fairness, undermining the benefits of efficient adaptation. To address this, we introduce Bias-Alleviating Low-Rank Adaptation (BA-LoRA). Our approach is founded on a principled decomposition of Catastrophic Inheritance into three core challenges: Knowledge Drift, Representation Collapse, and Overfitting to Noise. BA-LoRA systematically mitigates these issues by incorporating a trio of targeted regularizers: consistency, diversity, and an SVD-based term, designed to preserve core knowledge, promote representational richness, and encourage robust, low-rank output representations, respectively. We conduct comprehensive evaluations on a suite of Natural Language Generation (NLG) and Natural Language Understanding (NLU) tasks using diverse, prominent open-source language models (e.g., LLaMA-2-7B and DeBERTa-v3-base). Our results show that BA-LoRA not only outperforms state-of-the-art LoRA variants in terms of performance and stability, but also demonstrates superior robustness and bias mitigation on targeted evaluations. These results provide evidence that BA-LoRA can counteract the adverse effects of Catastrophic Inheritance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
