SimSCOOD：微调源代码模型中分布外泛化的系统性分析
SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models
AI总结：
本文提出SimSCOOD，首次系统模拟源代码数据的多种分布外场景，分析微调模型行为，发现LoRA微调在OOD泛化上始终显著优于全量微调。
摘要：
大规模代码数据集对于源代码模型的预训练已变得越来越容易获取。然而，在微调阶段，由于特定下游任务的特性和有限的标注资源，获取能够充分覆盖代码分布的具有代表性的训练数据仍然具有挑战性。此外，微调预训练模型可能导致遗忘先前获得的预训练知识。这些问题引发了分布外（OOD）泛化问题，并伴随意外的模型推理行为，而这些行为尚未得到系统性研究。在本文中，我们提出了首个系统性方法，沿源代码数据属性的不同维度模拟各种OOD场景，并研究微调模型在这些场景中的行为。我们考察了模型在不同微调方法下的行为，包括全量微调和低秩适配（LoRA）微调方法。我们在四个最先进的预训练模型上进行了全面分析，并将其应用于两个代码生成任务，揭示了多种归因于OOD泛化问题的失败模式。此外，我们的分析发现，在各种场景下，LoRA微调始终表现出显著优于全量微调的OOD泛化性能。
英文摘要：
Large code datasets have become increasingly accessible for pre-training source code models. However, for the fine-tuning phase, obtaining representative training data that fully covers the code distribution for specific downstream tasks remains challenging due to the task-specific nature and limited labeling resources. Moreover, fine-tuning pretrained models can result in forgetting previously acquired pre-training knowledge. These lead to out-of-distribution (OOD) generalization issues with unexpected model inference behaviors that have not been systematically studied yet. In this paper, we contribute the first systematic approach that simulates various OOD scenarios along different dimensions of source code data properties and study the fine-tuned model behaviors in such scenarios. We investigate the behaviors of models under different fine-tuning methodologies, including full fine-tuning and Low-Rank Adaptation (LoRA) fine-tuning methods. Our comprehensive analysis, conducted on four state-of-the-art pretrained models and applied to two code generation tasks, exposes multiple failure modes attributed to OOD generalization issues. Additionally, our analysis uncovers that LoRA fine-tuning consistently exhibits significantly better OOD generalization performance than full fine-tuning across various scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
