arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.04430cs.SEcs.AIcs.LG

大语言模型在跨语言代码克隆检测中的困境

The Struggles of LLMs in Cross-lingual Code Clone Detection

University of Luxembourg(卢森堡大学)

Micheline Bénédicte Moumoula, Abdoul Kader Kabore, Jacques Klein, Tegawendé Bissyande

AI总结:

本文评估了五种大语言模型和八种提示在跨语言代码克隆检测中的表现,发现其在复杂程序上效果有限,而基于嵌入模型训练的基础分类器在 XLCoST 和 CodeNet 数据集上分别比所有大语言模型高出约 1 个和 20 个百分点,表明嵌入表示更适合该任务。

摘要:

随着现代软件开发涉及多种编程语言,跨语言代码克隆检测在软件工程领域日益受到关注。大量研究已对该主题进行了探索,并提出了多种有前景的方法。受近年来机器学习领域重大进展的启发,尤其是大语言模型(LLMs)在各类任务中展现出的强大能力,本文重新审视了跨语言代码克隆检测问题。我们评估了五个(05)大语言模型和八种提示(08)在识别跨语言代码克隆方面的表现。此外,我们将这些结果与两种基线方法进行了比较。最后,我们评估了一个预训练嵌入模型,以检验其生成的表示在分类克隆对与非克隆对方面的有效性。涉及大语言模型和嵌入模型的研究均使用两个广泛使用的跨语言数据集 XLCoST 和 CodeNet 进行评估。我们的结果表明,对于简单的编程示例,大语言模型可以达到高达 0.99 的 F1 分数。然而,它们不仅在与复杂编程挑战相关的程序上表现较差,而且并不一定理解跨语言场景下“代码克隆”的含义。我们表明,使用嵌入模型将来自不同编程语言的代码片段表示在同一表示空间中,可以训练出一个基础分类器,该分类器在 XLCoST 和 CodeNet 数据集上分别比所有大语言模型高出约 1 个和约 20 个百分点。这一发现表明,尽管大语言模型展现出显著的能力,但嵌入模型提供的嵌入表示能够为实现跨语言代码克隆检测的最先进性能提供合适的表示。

英文摘要:

With the involvement of multiple programming languages in modern software development, cross-lingual code clone detection has gained traction within the software engineering community. Numerous studies have explored this topic, proposing various promising approaches. Inspired by the significant advances in machine learning in recent years, particularly Large Language Models (LLMs), which have demonstrated their ability to tackle various tasks, this paper revisits cross-lingual code clone detection. We evaluate the performance of five (05) LLMs and eight prompts (08) for the identification of cross-lingual code clones. Additionally, we compare these results against two baseline methods. Finally, we evaluate a pre-trained embedding model to assess the effectiveness of the generated representations for classifying clone and non-clone pairs. The studies involving LLMs and Embedding models are evaluated using two widely used cross-lingual datasets, XLCoST and CodeNet. Our results show that LLMs can achieve high F1 scores, up to 0.99, for straightforward programming examples. However, they not only perform less well on programs associated with complex programming challenges but also do not necessarily understand the meaning of "code clones" in a cross-lingual setting. We show that embedding models used to represent code fragments from different programming languages in the same representation space enable the training of a basic classifier that outperforms all LLMs by ~1 and ~20 percentage points on the XLCoST and CodeNet datasets, respectively. This finding suggests that, despite the apparent capabilities of LLMs, embeddings provided by embedding models offer suitable representations to achieve state-of-the-art performance in cross-lingual code clone detection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑