arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2308.04693cs.SEcs.IR
收藏

评估与优化神经机器翻译在支持代码检索模型中的有效性:一项基于CAT基准的研究

Evaluating and Optimizing the Effectiveness of Neural Machine Translation in Supporting Code Retrieval Models: A Study on the CAT Benchmark

Hung Phan, Ali Jannesari

AI总结:

本文在CAT基准上评估NMT自然语言到代码翻译性能,提出基于AST非终结节点子集的ASTTrans Representation表示方法,显著提升NMT效果,并使GraphCodeBERT和UniXcoder等先进代码搜索模型的MRR最高提升3.08%。

摘要:

神经机器翻译(NMT)在软件工程任务中得到了广泛应用。NMT在代码检索中的有效性依赖于其从源语言token序列学习到目标语言token序列的能力。虽然NMT在伪代码到代码的翻译中表现良好,但在新整理的、真实世界的代码文档/实现数据集中,从自然语言查询到源代码的翻译学习可能面临挑战。在本工作中,我们分析了NMT在新整理的CAT基准上进行自然语言到代码翻译的性能,该基准包含三个Java数据集TLCodeSum、CodeSearchNet、Funcom的优化版本,以及一个Python数据集PCSD。我们的评估表明,在该任务中,以CrystalBLEU和Meteor指标衡量,NMT的准确率较低。为了减轻NMT学习源代码复杂表示的任务负担,我们提出了ASTTrans Representation,这是一种利用非终结节点子集为抽象语法树(AST)定制的表示方法。我们表明,经典NMT方法在学习ASTTrans Representation方面显著优于学习代码token,Meteor分数提升最高达36%。此外,我们利用ASTTrans Representation,结合使用GraphCodeBERT和UniXcoder的最先进代码搜索流程进行组合代码搜索。我们学习ASTTrans Representation的NMT模型能够将这些最先进代码搜索流程的平均倒数排名(Mean Reciprocal Rank)提升最高达3.08%,并在CAT基准上改善23.08%的查询结果。

英文摘要:

Neural Machine Translation (NMT) is widely applied in software engineering tasks. The effectiveness of NMT for code retrieval relies on the ability to learn from the sequence of tokens in the source language to the sequence of tokens in the target language. While NMT performs well in pseudocode-to-code translation, it might have challenges in learning to translate from natural language query to source code in newly curated real-world code documentation/ implementation datasets. In this work, we analyze the performance of NMT in natural language-to-code translation in the newly curated CAT benchmark that includes the optimized versions of three Java datasets TLCodeSum, CodeSearchNet, Funcom, and a Python dataset PCSD. Our evaluation shows that NMT has low accuracy, measured by CrystalBLEU and Meteor metrics in this task. To alleviate the duty of NMT in learning complex representation of source code, we propose ASTTrans Representation, a tailored representation of an Abstract Syntax Tree (AST) using a subset of non-terminal nodes. We show that the classical approach NMT performs significantly better in learning ASTTrans Representation over code tokens with up to 36% improvement on Meteor score. Moreover, we leverage ASTTrans Representation to conduct combined code search processes from the state-of-the-art code search processes using GraphCodeBERT and UniXcoder. Our NMT models of learning ASTTrans Representation can boost the Mean Reciprocal Rank of these state-of-the-art code search processes by up to 3.08% and improve 23.08% of queries' results over the CAT benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑