解码器-only大语言模型是否是代码搜索的银弹?
Are Decoder-Only Large Language Models the Silver Bullet for Code Search?
AI总结:
本文评估了11种解码器-only LLM在代码搜索任务中的性能,发现经过微调的模型在零样本和微调设置下均优于编码器-only模型,且模型大小和训练数据组成对性能有显著影响。
摘要:
代码搜索对于代码重用至关重要,允许开发者高效地找到相关代码片段。强大的解码器-only大语言模型(LLMs)已革新了许多代码智能任务。然而,它们在基于检索的代码搜索任务中的有效性,特别是与已建立的编码器-based模型相比,仍缺乏深入研究。本文通过系统评估11种解码器-only LLM,分析其在零样本和微调设置下的性能。我们的结果表明,经过微调的解码器-only模型,特别是CodeGemma,在CoSQA$^+$基准上比编码器-only模型如UniXcoder高出40.4%的平均平均精度(MAP)。我们的分析进一步揭示了两个关键要点:首先,模型大小与性能之间的关系是非单调的,中等大小的模型通常优于更大的变体;其次,训练数据的组成至关重要,多语言数据集有助于泛化,而少量特定语言的数据可能成为噪声并干扰模型效果。这些发现为选择和优化现代LLM用于代码搜索提供了全面的指导。
英文摘要:
Code search is essential for code reuse, allowing developers to efficiently locate relevant code snippets. The advent of powerful decoder-only Large Language Models (LLMs) has revolutionized many code intelligence tasks. However, their effectiveness for the retrieval-based task of code search, particularly compared to established encoder-based models, remains underexplored. This paper addresses this gap by presenting a large-scale systematic evaluation of eleven decoder-only LLMs, analyzing their performance across zero-shot and fine-tuned settings. Our results show that fine-tuned decoder-only models, particularly CodeGemma, significantly outperform encoder-only models like UniXcoder, achieving a 40.4% higher Mean Average Precision (MAP) on the CoSQA$^+$ benchmark. Our analysis further reveals two crucial nuances for practitioners: first, the relationship between model size and performance is non-monotonic, with mid-sized models often outperforming larger variants; second, the composition of the training data is critical, as a multilingual dataset enhances generalization while a small amount of data from a specific language can act as noise and interfere with model effectiveness. These findings offer a comprehensive guide to selecting and optimizing modern LLMs for code search.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。