构建高质量训练数据集对神经代码搜索的重要性
On the Importance of Building High-quality Training Datasets for Neural Code Search
摘要：
神经代码搜索的性能受到神经模型所源自的训练数据的质量的显著影响。为了建立从自然语言到程序设计语言的精确映射，需要大量高质量的查询和代码对。由于可用性有限，大多数广泛使用的代码搜索数据集都是在折衷的情况下建立的，例如使用代码注释作为查询的替代。我们对一个著名的代码搜索数据集的实证研究表明，超过三分之一的查询包含噪音，使他们偏离自然用户查询。通过噪声数据训练的模型在应用于现实世界场景时面临严重的性能下降。提高数据集质量，使其样本的查询与真实用户查询语义一致，是神经代码搜索实用性的关键。在本文中，我们提出了一个数据清洗框架，包括两个后续的过滤器：基于规则的语法过滤器和基于模型的语义过滤器。这是第一个将语义查询清理应用于代码搜索数据集的框架。实验上，我们评估了我们的框架的有效性，两个广泛使用的代码搜索模型和三个手动注释的代码检索基准。使用我们框架中过滤后的数据集训练流行的DeepCS模型，其性能平均提高了19.2%的MRR和21.3%的MRR@1，这是三个验证基准测试的平均值。
英文摘要：
The performance of neural code search is significantly influenced by the quality of the training data from which the neural models are derived. A large corpus of high-quality query and code pairs is demanded to establish a precise mapping from the natural language to the programming language. Due to the limited availability, most widely-used code search datasets are established with compromise, such as using code comments as a replacement of queries. Our empirical study on a famous code search dataset reveals that over one-third of its queries contain noises that make them deviate from natural user queries. Models trained through noisy data are faced with severe performance degradation when applied in real-world scenarios. To improve the dataset quality and make the queries of its samples semantically identical to real user queries is critical for the practical usability of neural code search. In this paper, we propose a data cleaning framework consisting of two subsequent filters: a rule-based syntactic filter and a model-based semantic filter. This is the first framework that applies semantic query cleaning to code search datasets. Experimentally, we evaluated the effectiveness of our framework on two widely-used code search models and three manually-annotated code retrieval benchmarks. Training the popular DeepCS model with the filtered dataset from our framework improves its performance by 19.2% MRR and 21.3% Answer@1, on average with the three validation benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
