arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.10808cs.SEcs.AI
收藏

PELLI:一种有效整合LLMs进行高质量软件生成的框架

PELLI: Framework to effectively integrate LLMs for quality software generation

Department of Digitalization, Copenhagen Business School(数字化部门,哥本哈根商学院)

Rasmus Krebs, Somnath Mazumdar

AI总结:

PELLI框架通过迭代分析提升LLMs在软件生成中的质量,评估五个LLMs在可维护性、性能和可靠性上的表现,揭示提示设计对代码质量的影响。

摘要:

近期研究表明,当LLMs得到适当提示和配置时,它们的表现往往混合且有时可以达到或超过基线水平。然而,这些比较存在两个主要问题。首先,它们大多只将可靠性作为比较指标,并选择了少数LLMs(如Codex和ChatGPT)进行比较。本文提出了一种综合的代码质量评估框架,称为通过LLM迭代的程序性卓越(PELLI)。PELLI是一种基于迭代分析的过程,旨在保持高质量的代码更改。我们通过全面评估扩展了最先进的方法,生成用于分析三个主要非功能性需求(如可维护性、性能和可靠性)的定量指标,同时选择了五种流行的LLMs。为了确保PELLI的适用性,我们选择了三个应用领域,并遵循Python编码标准。遵循这一框架,从业者可以确保LLMs与人类开发者之间的和谐整合,确保其潜力得到充分发挥。PELLI可以为开发者提供实用指南,帮助他们在遵循公认质量标准的同时利用LLMs。本研究的成果对于推进LLMs在现实应用中的技术发展至关重要,为相关利益相关者提供了清晰的理解,即这些LLMs在哪些方面表现优异,哪些方面需要进一步改进。总体而言,基于三个非功能性需求,我们发现GPT-4T和Gemini略优于其他模型。我们还发现,提示设计可以影响整体代码质量。此外,每个应用领域在各种指标上均表现出高分和低分,甚至在同一指标上不同提示也会产生差异。

英文摘要:

Recent studies have revealed that when LLMs are appropriately prompted and configured, they demonstrate mixed results. Such results often meet or exceed the baseline performance. However, these comparisons have two primary issues. First, they mostly considered only reliability as a comparison metric and selected a few LLMs (such as Codex and ChatGPT) for comparision. This paper proposes a comprehensive code quality assessment framework called Programmatic Excellence via LLM Iteration (PELLI). PELLI is an iterative analysis-based process that upholds high-quality code changes. We extended the state-of-the-art by performing a comprehensive evaluation that generates quantitative metrics for analyzing three primary nonfunctional requirements (such as maintainability, performance, and reliability) while selecting five popular LLMs. For PELLI's applicability, we selected three application domains while following Python coding standards. Following this framework, practitioners can ensure harmonious integration between LLMs and human developers, ensuring that their potential is fully realized. PELLI can serve as a practical guide for developers aiming to leverage LLMs while adhering to recognized quality standards. This study's outcomes are crucial for advancing LLM technologies in real-world applications, providing stakeholders with a clear understanding of where these LLMs excel and where they require further refinement. Overall, based on three nonfunctional requirements, we have found that GPT-4T and Gemini performed slightly better. We also found that prompt design can influence the overall code quality. In addition, each application domain demonstrated high and low scores across various metrics, and even within the same metrics across different prompts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑