arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.22240cs.SE
收藏

解码器-only大语言模型是否是代码搜索的银弹?

Are Decoder-Only Large Language Models the Silver Bullet for Code Search?

School of Software Engineering, Sun Yat-sen University(中山大学软件学院);Zhuhai Key Laboratory of Trusted Large Language Models(珠海可信大模型重点实验室);Shenzhen Research Institute of Sun Yat-sen University(中山大学深圳研究院)

Yuxuan Chen, Mingwei Liu, Guangsheng Ou, Anji Li, Dekun Dai, Yanlin Wang, Zibin Zheng

AI总结:

本文评估了11种解码器-only LLM在代码搜索任务中的性能,发现经过微调的模型在零样本和微调设置下均优于编码器-only模型,且模型大小和训练数据组成对性能有显著影响。

摘要:

代码搜索对于代码重用至关重要,允许开发者高效地找到相关代码片段。强大的解码器-only大语言模型(LLMs)已革新了许多代码智能任务。然而,它们在基于检索的代码搜索任务中的有效性,特别是与已建立的编码器-based模型相比,仍缺乏深入研究。本文通过系统评估11种解码器-only LLM,分析其在零样本和微调设置下的性能。我们的结果表明,经过微调的解码器-only模型,特别是CodeGemma,在CoSQA$^+$基准上比编码器-only模型如UniXcoder高出40.4%的平均平均精度(MAP)。我们的分析进一步揭示了两个关键要点:首先,模型大小与性能之间的关系是非单调的,中等大小的模型通常优于更大的变体;其次,训练数据的组成至关重要,多语言数据集有助于泛化,而少量特定语言的数据可能成为噪声并干扰模型效果。这些发现为选择和优化现代LLM用于代码搜索提供了全面的指导。

英文摘要:

Code search is essential for code reuse, allowing developers to efficiently locate relevant code snippets. The advent of powerful decoder-only Large Language Models (LLMs) has revolutionized many code intelligence tasks. However, their effectiveness for the retrieval-based task of code search, particularly compared to established encoder-based models, remains underexplored. This paper addresses this gap by presenting a large-scale systematic evaluation of eleven decoder-only LLMs, analyzing their performance across zero-shot and fine-tuned settings. Our results show that fine-tuned decoder-only models, particularly CodeGemma, significantly outperform encoder-only models like UniXcoder, achieving a 40.4% higher Mean Average Precision (MAP) on the CoSQA$^+$ benchmark. Our analysis further reveals two crucial nuances for practitioners: first, the relationship between model size and performance is non-monotonic, with mid-sized models often outperforming larger variants; second, the composition of the training data is critical, as a multilingual dataset enhances generalization while a small amount of data from a specific language can act as noise and interfere with model effectiveness. These findings offer a comprehensive guide to selecting and optimizing modern LLMs for code search.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑