arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.03031cs.SE
收藏

使用开源大语言模型进行调试:一项评估

Debugging with Open-Source Large Language Models: An Evaluation

Yacine Majdoub, Eya Ben Charrada

AI总结:

本文初步评估了五个开源大语言模型在DebugBench基准上的代码调试能力,发现DeepSeek-Coder在Python、Java和C++上均表现最佳,得分范围为43.9%至66.6%。

摘要:

大语言模型在支持软件开发任务方面已展现出良好潜力。因此,越来越多的开发者转向LLM(如ChatGPT)来帮助他们修复有缺陷的代码。虽然这可以节省时间和精力,但许多公司由于严格的代码共享政策而禁止使用。为解决这一问题,公司可以在本地运行开源LLM。但迄今为止,评估开源大语言模型在调试方面性能的研究并不多。本工作是对开源LLM修复有缺陷代码能力的初步评估。该评估涵盖五个开源大语言模型,并使用基准DebugBench,其中包含超过4000个用Python、Java和C++编写的有缺陷代码实例。开源LLM取得的得分范围为43.9%至66.6%,其中DeepSeek-Coder在三种编程语言中均取得了最佳得分。

英文摘要:

Large language models have shown good potential in supporting software development tasks. This is why more and more developers turn to LLMs (e.g. ChatGPT) to support them in fixing their buggy code. While this can save time and effort, many companies prohibit it due to strict code sharing policies. To address this, companies can run open-source LLMs locally. But until now there is not much research evaluating the performance of open-source large language models in debugging. This work is a preliminary evaluation of the capabilities of open-source LLMs in fixing buggy code. The evaluation covers five open-source large language models and uses the benchmark DebugBench which includes more than 4000 buggy code instances written in Python, Java and C++. Open-source LLMs achieved scores ranging from 43.9% to 66.6% with DeepSeek-Coder achieving the best score for all three programming languages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑