评估代码摘要技术:一种新度量与实证表征
Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization
AI总结:
本文针对代码摘要评估中现有自动度量依赖参考摘要的缺陷,提出一种基于对比学习的新度量,以捕捉生成摘要与代码语义的一致性,并通过实证验证其能更有效表征开发人员评估。
摘要:
文献中已提出多种代码摘要技术,用于自动记录代码片段或函数。理想情况下,软件开发人员应参与评估生成摘要的质量。然而,在大多数情况下,研究人员依赖自动评估度量,如BLEU、ROUGE和METEOR。这些度量都基于同一假设:生成摘要与开发人员编写的参考摘要之间的文本相似度越高,其质量越高。然而,这一假设存在两个不足之处:(i)参考摘要(例如通过挖掘软件仓库收集的代码注释)可能质量较低甚至过时;(ii)生成摘要虽然措辞与参考摘要不同,但可能在语义上等价,因此仍适合记录代码片段。在本文中,我们对不同类型度量在捕捉生成摘要质量方面的互补性进行了彻底的实证研究。此外,我们提出通过考虑一个新维度来解决现有度量的局限性,该维度捕捉生成摘要与所记录代码片段语义的一致性程度,独立于参考摘要。为此,我们提出一种基于对比学习的新度量来捕捉上述方面。我们通过实证表明,纳入这一新维度能够更有效地表征开发人员对自动生成摘要质量的评估。
英文摘要:
Several code summarization techniques have been proposed in the literature to automatically document a code snippet or a function. Ideally, software developers should be involved in assessing the quality of the generated summaries. However, in most cases, researchers rely on automatic evaluation metrics such as BLEU, ROUGE, and METEOR. These metrics are all based on the same assumption: The higher the textual similarity between the generated summary and a reference summary written by developers, the higher its quality. However, there are two reasons for which this assumption falls short: (i) reference summaries, e.g., code comments collected by mining software repositories, may be of low quality or even outdated; (ii) generated summaries, while using a different wording than a reference one, could be semantically equivalent to it, thus still being suitable to document the code snippet. In this paper, we perform a thorough empirical investigation on the complementarity of different types of metrics in capturing the quality of a generated summary. Also, we propose to address the limitations of existing metrics by considering a new dimension, capturing the extent to which the generated summary aligns with the semantics of the documented code snippet, independently from the reference summary. To this end, we present a new metric based on contrastive learning to capture said aspect. We empirically show that the inclusion of this novel dimension enables a more effective representation of developers' evaluations regarding the quality of automatically generated summaries.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。