arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.23340cs.SE
收藏

LLMs能否成为有效的代码贡献者?对开源项目的研究

Can LLMs be Effective Code Contributors? A Study on Open-source Projects

New Jersey Institute of Technology(新泽西理工学院)

Chun Jie Chong, Muyeed Ahmed, Zhihao, Yao, Iulian Neamtiu

AI总结:

本文研究了LLMs在开源项目中的代码贡献效果,通过评估框架发现LLMs在修复bug和添加功能时成功率低,存在语法错误和测试失败等问题,主要贡献是提出评估方法和改进建议。

摘要:

LLM生成的代码被广泛使用,其在生产代码中的贡献份额预计会增加。然而,我们尚未达到LLMs能有效贡献生产代码的阶段。本文提出了一种方法,通过验证和验证来评估LLM在修复或添加功能到现有项目中的适用性,并提出了建议。研究对象是大规模开源项目,如FFmpeg和wolfSSL。首先,我们开发了一个框架,利用验证和验证来评估给定LLM在修复或添加功能到现有项目中的适用性。其次,我们将该框架应用于八个流行开源项目和三个LLM(GPT-4o、Minstral3和Qwen3-Coder)的212个提交(bug修复和小型功能改进)。成功率从0%到60%不等,取决于项目。LLMs在多种方式上失败,从生成语法错误的代码,到生成无法通过基本(静态)验证或通过项目测试套件验证的代码。特别是,LLMs在生成新代码、处理超出一定范围的上下文(函数或文件)以及在许多情况下其成功归因于鹦鹉学舌训练过的代码更改。

英文摘要:

LLM-generated code is widely used, and the share of committed code produced by LLMs is expected to increase. However, we are not at a point where LLMs can be effective contributors to production code. We present an approach that exposes the shortcomings of LLM generation on such projects, and proposes recommendations; the targets of our study are sizable open-source projects, e.g., FFmpeg and wolfSSL. First, we developed a framework that uses verification and validation to evaluate a given LLM's suitability to fix or add features to an existing project. Second, we apply the framework to 212 commits (bug fixes and small feature improvements) in eight popular open-source projects and three LLMs: GPT-4o, Ministral3, and Qwen3-Coder. The success rate varied from 0% to 60% depending on the project. The LLMs failed in a variety of ways, from generating syntactically incorrect code, to producing code that fails basic (static) verification, or validation via the project's test suite. In particular, the LLMs struggle with generating new code, handling contexts (function or file) outside a certain size range, and in many cases their success is due to parroting code changes they have been trained on.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑