arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.04030cs.SEcs.AI
收藏

CrossCodeBench:Source Code模型的基准跨任务推广

CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models

Nanjing University(南京大学);University of Texas at Dallas(德克萨斯大学达拉斯分校)

Changan Niu, Chuanyi Li, Vincent Ng, Bin Luo

摘要:

尽管最近的进展表明,在大规模源代码数据上预训练的模型能够获得可观的泛化能力,但它仍然需要目标任务的大量数据进行微调。模型泛化的有效性在很大程度上受到微调数据的大小和质量的影响,这对于资源有限或不可用的目标任务是不利的。因此,跨任务泛化,旨在提高模型对未知任务的泛化能力,具有很强的研究和应用价值。在本文中,我们提出了一个大规模的基准测试,其中包括216个现有的代码相关的任务。然后,我们用相应的Meta信息(如任务描述和指令)对每个任务进行注释,其中包含有关任务的详细信息和解决方案指南。这也有助于我们轻松地创建各种各样的“训练/评估”任务分割,以评估模型的各种跨任务泛化能力。通过初步实验证明,少样本学习和任务指令学习等上下文学习方法可以显著提高模型的跨任务泛化能力,从而展示了在该基准上进行跨任务学习研究的良好前景。我们希望数据集的收集和我们的基准将促进未来的工作,而不仅限于跨任务的泛化。

英文摘要:

Despite the recent advances showing that a model pre-trained on large-scale source code data is able to gain appreciable generalization capability, it still requires a sizeable amount of data on the target task for fine-tuning. And the effectiveness of the model generalization is largely affected by the size and quality of the fine-tuning data, which is detrimental for target tasks with limited or unavailable resources. Therefore, cross-task generalization, with the goal of improving the generalization of the model to unseen tasks that have not been seen before, is of strong research and application value. In this paper, we propose a large-scale benchmark that includes 216 existing code-related tasks. Then, we annotate each task with the corresponding meta information such as task description and instruction, which contains detailed information about the task and a solution guide. This also helps us to easily create a wide variety of ``training/evaluation'' task splits to evaluate the various cross-task generalization capabilities of the model. Then we perform some preliminary experiments to demonstrate that the cross-task generalization of models can be largely improved by in-context learning methods such as few-shot learning and learning from task instructions, which shows the promising prospects of conducting cross-task learning research on our benchmark. We hope that the collection of the datasets and our benchmark will facilitate future work that is not limited to cross-task generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑