arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2108.09619cs.SEcs.LG
收藏

评估方法对代码摘要的影响

Impact of Evaluation Methodologies on Code Summarization

The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Pengyu Nie, Jiyang Zhang, Junyi Jessy Li, Raymond J. Mooney, Milos Gligoric

摘要:

人们对开发用于代码摘要任务的机器学习(ML)模型越来越感兴趣,例如,注释生成和方法命名。尽管ML模型的有效性大幅提高,但评估方法,即,人们将数据集划分为训练集、验证集和测试集的方式还没有得到很好的研究。具体地说,以前的代码总结工作没有考虑评估期间代码和注释的时间戳。这可能导致评估与预期用例不一致。在本文中,我们介绍了时间分段的评估方法,这是新的代码摘要研究社区,并比较它与混合项目和跨项目的方法,已被普遍使用。每种方法都可以映射到一些用例,在评估用于代码摘要的ML模型时应该采用时间分段方法。为了评估方法的影响,我们收集了一个带有时间戳的(代码,注释)对数据集,以训练和评估几个最近的ML模型用于代码总结。我们的实验表明,不同的方法会导致相互矛盾的评估结果。我们邀请社区扩大评价中使用的一套方法。

英文摘要:

There has been a growing interest in developing machine learning (ML) models for code summarization tasks, e.g., comment generation and method naming. Despite substantial increase in the effectiveness of ML models, the evaluation methodologies, i.e., the way people split datasets into training, validation, and test sets, were not well studied. Specifically, no prior work on code summarization considered the timestamps of code and comments during evaluation. This may lead to evaluations that are inconsistent with the intended use cases. In this paper, we introduce the time-segmented evaluation methodology, which is novel to the code summarization research community, and compare it with the mixed-project and cross-project methodologies that have been commonly used. Each methodology can be mapped to some use cases, and the time-segmented methodology should be adopted in the evaluation of ML models for code summarization. To assess the impact of methodologies, we collect a dataset of (code, comment) pairs with timestamps to train and evaluate several recent ML models for code summarization. Our experiments show that different methodologies lead to conflicting evaluation results. We invite the community to expand the set of methodologies used in evaluations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑