过多的幅度有多大？探索LoRA秩的权衡以保留知识和领域鲁棒性
How Much is Too Much? Exploring LoRA Rank Trade-offs for Retaining Knowledge and Domain Robustness
AI总结：
本文探讨LoRA秩配置在保留知识和领域鲁棒性方面的权衡，通过多任务评估展示LoRA在推理任务中的竞争力。
摘要：
大型语言模型越来越通过微调适应下游任务。全监督微调（SFT）和参数高效微调（PEFT）方法，如低秩适应（LoRA），是两种主导方法。虽然PEFT方法因其计算效率而被广泛使用，但其配置（例如秩）的影响在下游问答任务和泛化方面仍缺乏深入研究。在本工作中，我们在多个推理和回忆数据集上进行了全面评估，进行秩扫描以量化SFT和PEFT之间的权衡。我们还比较了PEFT和SFT模型在领域内和领域外适应中的准确性，突显了不同的泛化行为和任务特定的遗忘。我们证明LoRA在某些特定秩值上能够实现具有竞争力甚至更优的性能。此外，我们通过谱特征和层间注意力结构分析内部表示，提供了关于表示漂移和注意力模式结构变化的见解。
英文摘要：
Large language models are increasingly adapted to downstream tasks through fine-tuning. Full supervised fine-tuning (SFT) and parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), are two dominant approaches. While PEFT methods are widely used for their computational efficiency, the implications of their configurations (e.g., rank) remain under-explored in downstream Q&A tasks and generalisation. In this work, we perform a comprehensive evaluation across multiple reasoning and recall datasets, conducting a rank sweep to quantify the trade-off between SFT and PEFT. We also compare the accuracy of PEFT and SFT models across in-domain and out-of-domain adaptation, highlighting distinct generalisation behaviour and task-specific forgetting. We demonstrate that LoRA achieves competitive and in some cases superior performance compared to SFT, particularly on reasoning tasks at specific rank values. Additionally, we analyze the internal representations via spectral features and layer-wise attention structures, offering insights into representational drift and structural changes in attention patterns.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
