SparseCoder:面向文件级代码摘要的标识符感知稀疏Transformer
SparseCoder: Identifier-Aware Sparse Transformer for File-Level Code Summarization
AI总结:
针对文件级长代码摘要任务中Transformer因计算复杂度高难以处理长输入的问题,提出标识符感知稀疏Transformer模型SparseCoder,结合滑动窗口与两类稀疏注意力捕获长短依赖,在自建Python数据集FILE-CS上取得最优性能且内存开销低。
摘要:
代码摘要旨在生成源代码的自然语言描述,帮助程序员快速理解和维护代码。以往的代码摘要研究主要聚焦于方法级,本文则研究文件级代码摘要,该任务可辅助程序员理解和维护大型源代码项目。与方法级代码摘要不同,文件级代码摘要通常涉及单个文件内的长源代码,这给基于Transformer的模型理解代码语义带来挑战——由于计算复杂度随输入序列长度呈二次方增长,这类模型的最大输入长度难以设置到足以良好处理长代码输入的较大数值。为应对这一挑战,我们提出SparseCoder,一种可有效处理长代码序列的标识符感知稀疏Transformer。具体而言,SparseCoder采用滑动窗口自注意力机制建模短期依赖,并利用代码的结构信息,通过引入名为全局注意力和标识符注意力的两种稀疏注意力模式,捕获源代码标识符之间的长期依赖。为评估SparseCoder的性能,我们构建了一个用于Python文件级代码摘要的新数据集FILE-CS。实验结果表明,与包括全自注意力模型和稀疏模型在内的其他预训练模型相比,我们的SparseCoder模型取得了当前最优性能。此外,我们的模型内存开销低,且能达到与使用全自注意力机制的模型相当的性能。
英文摘要:
Code summarization aims to generate natural language descriptions of source code, facilitating programmers to understand and maintain it rapidly. While previous code summarization efforts have predominantly focused on method-level, this paper studies file-level code summarization, which can assist programmers in understanding and maintaining large source code projects. Unlike method-level code summarization,file-level code summarization typically involves long source code within a single file, which makes it challenging for Transformer-based models to understand the code semantics for the maximum input length of these models is difficult to set to a large number that can handle long code input well, due to the quadratic scaling of computational complexity with the input sequence length. To address this challenge, we propose SparseCoder, an identifier-aware sparse transformer for effectively handling long code sequences. Specifically, the SparseCoder employs a sliding window mechanism for self-attention to model short-term dependencies and leverages the structure message of code to capture long-term dependencies among source code identifiers by introducing two types of sparse attention patterns named global and identifier attention. To evaluate the performance of SparseCoder, we construct a new dataset FILE-CS for file-level code summarization in Python. Experimental results show that our SparseCoder model achieves state-of-the-art performance compared with other pre-trained models, including full self-attention and sparse models. Additionally, our model has low memory overhead and achieves comparable performance with models using full self-attention mechanism.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。