arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.11702cs.SEcs.AIcs.LG
收藏

ChatGPT 能否支持开发者?大型语言模型代码生成能力的实证评估

Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation

York University(约克大学)

Kailun Jin, Chung-Yu Wang, Hung Viet Pham, Hadi Hemmati

AI总结:

本研究通过分析DevGPT中开发者与ChatGPT的对话,实证评估了LLM代码生成在现实中的使用,发现其主要用于概念展示和文档示例,而非生产代码,指出未来改进方向。

摘要:

大型语言模型(LLMs)在代码生成方面展现了显著的能力,许多先前的研究表明它们在各种开发场景中具有广阔的应用前景。然而,这些研究主要是在研究环境中进行评估,这导致在理解LLMs如何在现实世界中有效支持开发者方面存在显著空白。为了解决这一问题,我们对DevGPT中的对话进行了实证分析,DevGPT是一个从开发者与ChatGPT的对话中收集的数据集(通过GitHub等平台上的共享链接功能捕获)。我们的实证发现表明,当前使用LLM生成代码的实践通常仅限于展示高层概念或提供文档中的示例,而非用作生产就绪代码。这些发现表明,在LLMs成为现代软件开发不可或缺的一部分之前,还需要大量未来工作来改进其代码生成能力。

英文摘要:

Large language models (LLMs) have demonstrated notable proficiency in code generation, with numerous prior studies showing their promising capabilities in various development scenarios. However, these studies mainly provide evaluations in research settings, which leaves a significant gap in understanding how effectively LLMs can support developers in real-world. To address this, we conducted an empirical analysis of conversations in DevGPT, a dataset collected from developers' conversations with ChatGPT (captured with the Share Link feature on platforms such as GitHub). Our empirical findings indicate that the current practice of using LLM-generated code is typically limited to either demonstrating high-level concepts or providing examples in documentation, rather than to be used as production-ready code. These findings indicate that there is much future work needed to improve LLMs in code generation before they can be integral parts of modern software development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑