arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.17166cs.SEcs.CL
收藏

基于NLP的软件工程模型评价研究

On the Evaluation of NLP-based Models for Software Engineering

Delft University of Technology(代尔夫特理工大学);University of Tehran(德黑兰大学)

Maliheh Izadi, Matin Nili Ahmadabadi

摘要:

基于NLP的模型越来越多地被纳入解决SE问题。这些模型要么在SE领域中使用,几乎没有变化,要么根据源代码及其独特的特性进行了大量的定制。其中许多方法被认为优于或补充了现有的解决方案。然而,这里出现了一个重要的问题:“这些模型在SE社区中是否得到了公平和一致的评估?".为了回答这个问题,我们回顾了研究人员如何评估基于NLP的SE问题模型。调查结果表明,目前没有一致的和广泛接受的协议,这些模型的评价。虽然在不同的研究中评估了同一任务的不同方面,但度量标准是基于自定义选择而不是系统定义的,最后,答案是逐案收集和解释的。因此,迫切需要提供一种评估基于NLP的模型的方法,以进行一致的评估,并保持公平和有效比较的可能性。

英文摘要:

NLP-based models have been increasingly incorporated to address SE problems. These models are either employed in the SE domain with little to no change, or they are greatly tailored to source code and its unique characteristics. Many of these approaches are considered to be outperforming or complementing existing solutions. However, an important question arises here: "Are these models evaluated fairly and consistently in the SE community?". To answer this question, we reviewed how NLP-based models for SE problems are being evaluated by researchers. The findings indicate that currently there is no consistent and widely-accepted protocol for the evaluation of these models. While different aspects of the same task are being assessed in different studies, metrics are defined based on custom choices, rather than a system, and finally, answers are collected and interpreted case by case. Consequently, there is a dire need to provide a methodological way of evaluating NLP-based models to have a consistent assessment and preserve the possibility of fair and efficient comparison.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑