LoRA Dropout：一种用于过拟合控制的稀疏正则化方法
LoRA Dropout as a Sparsity Regularizer for Overfitting Control
AI总结：
针对LoRA类参数高效微调方法的过拟合问题，提出LoRA Dropout稀疏正则化机制，推导泛化误差界并设计测试时集成策略，经多NLP任务实验验证可提升模型准确率与校准效果。
摘要：
以LoRA（低秩适配）为代表的参数高效微调方法在使大规模预训练模型适配下游任务中发挥着关键作用。然而，微调LoRA系列模型同样面临在训练数据集上过拟合的风险，而目前针对基于LoRA的参数高效微调（PEFT）方法，仍缺乏控制过拟合的理论指导与实用机制。\n本文提出了一种适用于基于LoRA方法的LoRA Dropout机制，通过向可学习低秩矩阵引入随机噪声并提升参数稀疏性来实现这一目标。随后，我们从稀疏正则化的角度论证了LoRA Dropout机制的理论原理，并给出了该框架下的泛化误差界。理论结果表明，适当的稀疏性有助于缩小经验风险与泛化风险之间的差距，从而控制过拟合。\n此外，基于LoRA Dropout框架，我们提出了一种测试时集成策略，并提供了理论依据证明该集成方法能够进一步压缩误差界，在推理阶段带来更优的性能。在多种自然语言处理（NLP）任务上开展的大量实验，为LoRA Dropout框架在提升模型准确率与校准效果方面的有效性提供了实际验证。
英文摘要：
Parameter-efficient fine-tuning methods, represented by LoRA, play an essential role in adapting large-scale pre-trained models to downstream tasks. However, fine-tuning LoRA-series models also faces the risk of overfitting on the training dataset, and yet there's still a lack of theoretical guidance and practical mechanism to control overfitting on LoRA-based PEFT methods. In this paper, we propose a LoRA Dropout mechanism for the LoRA-based methods by introducing random noises to the learnable low-rank matrices and increasing parameter sparsity. We then demonstrate the theoretical mechanism of our LoRA Dropout mechanism from the perspective of sparsity regularization by providing a generalization error bound under this framework. Theoretical results show that appropriate sparsity would help tighten the gap between empirical and generalization risks and thereby control overfitting. Furthermore, based on the LoRA Dropout framework, we introduce a test-time ensemble strategy and provide theoretical evidence demonstrating that the ensemble method can further compress the error bound, and lead to better performance during inference time. Extensive experiments on various NLP tasks provide practical validations of the effectiveness of our LoRA Dropout framework in improving model accuracy and calibration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
