arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.08018cs.SEcs.AIcs.CL
收藏

大语言模型在代码摘要任务上的性能分析

Analyzing the Performance of Large Language Models on Code Summarization

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Rajarshi Haldar, Julia Hockenmaier

AI总结:

该研究分析大语言模型在代码摘要任务的性能,发现其表现与代码和参考描述的词元重叠度相关,重叠主要集中在函数名,且BLEU、BERTScore等评估指标高度相关,难提供额外评估价值。

摘要:

Llama 2等大语言模型(LLMs)在涉及自然语言与源代码的任务上表现优异,尤其是代码摘要和代码生成任务。我们发现,在代码摘要任务中,这些模型在单个样本上的性能通常取决于数据集中代码与对应参考自然语言描述之间的(子词)词元重叠程度。这种词元重叠的产生,是因为标准数据集中的参考描述(对应大型代码库中的文档字符串)往往与其所描述的函数名称高度相似。我们还发现,这种词元重叠主要出现在代码的函数名称中,并对比了移除函数名称与移除代码结构两种情况下这些模型的相对性能。此外我们还表明,使用BLEU、BERTScore等多种评估指标几乎无法带来额外的洞察,因为这些指标之间具有高度相关性。

英文摘要:

Large language models (LLMs) such as Llama 2 perform very well on tasks that involve both natural language and source code, particularly code summarization and code generation. We show that for the task of code summarization, the performance of these models on individual examples often depends on the amount of (subword) token overlap between the code and the corresponding reference natural language descriptions in the dataset. This token overlap arises because the reference descriptions in standard datasets (corresponding to docstrings in large code bases) are often highly similar to the names of the functions they describe. We also show that this token overlap occurs largely in the function names of the code and compare the relative performance of these models after removing function names versus removing code structure. We also show that using multiple evaluation metrics like BLEU and BERTScore gives us very little additional insight since these metrics are highly correlated with each other.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑