评论“翻译”的代码：数据、分析、基线和评估
Code to Comment "Translation": Data, Metrics, Baselining & Evaluation
摘要：
注释与代码的关系，特别是在给定代码的情况下生成有用注释的任务，一直是人们感兴趣的。最早的方法是基于强大的句法理论的评论结构，并依赖于文本模板。最近，研究人员已经将深度学习方法应用于这项任务，具体来说，可训练的生成翻译模型已知对于自然语言翻译非常有效（例如，从德语到英语）。我们仔细研究了这里的基本假设：生成评论的任务与自然语言之间的翻译任务非常相似，因此可以使用类似的模型和评估指标。我们分析了几个最近的代码注释数据集：CodeNN，DeepCom，FunCom和DocString。我们将它们与WMT 19进行比较，WMT 19是一个标准数据集，经常用于训练最先进的自然语言翻译人员。我们发现代码注释数据和WMT 19自然语言数据之间存在一些有趣的差异。接下来，我们描述并进行一些研究来校准BLEU（通常用作评论质量的衡量标准）。使用来自不同项目、同一项目、同一类等的方法的“亲和对”;我们的研究表明，某些数据集的当前性能可能需要大幅提高。我们还认为，相当天真的信息检索（IR）的方法做得足够好，在这项任务被认为是一个合理的基线。最后，我们提出了一些建议，我们的研究结果可能会在未来的研究在这方面。
英文摘要：
The relationship of comments to code, and in particular, the task of generating useful comments given the code, has long been of interest. The earliest approaches have been based on strong syntactic theories of comment-structures, and relied on textual templates. More recently, researchers have applied deep learning methods to this task, and specifically, trainable generative translation models which are known to work very well for Natural Language translation (e.g., from German to English). We carefully examine the underlying assumption here: that the task of generating comments sufficiently resembles the task of translating between natural languages, and so similar models and evaluation metrics could be used. We analyze several recent code-comment datasets for this task: CodeNN, DeepCom, FunCom, and DocString. We compare them with WMT19, a standard dataset frequently used to train state of the art natural language translators. We found some interesting differences between the code-comment data and the WMT19 natural language data. Next, we describe and conduct some studies to calibrate BLEU (which is commonly used as a measure of comment quality). using "affinity pairs" of methods, from different projects, in the same project, in the same class, etc; Our study suggests that the current performance on some datasets might need to be improved substantially. We also argue that fairly naive information retrieval (IR) methods do well enough at this task to be considered a reasonable baseline. Finally, we make some suggestions on how our findings might be used in future research in this area.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
