arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.03462cs.SE
收藏

迈向应用级代码生成的功能性和非功能性评估

Toward Functional and Non-Functional Evaluation of Application-Level Code Generation

Chongqing University(重庆大学);Peking University(北京大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))

Ruwei Pan, Yakun Zhang, Qingyuan Liang, Yueheng Zhu, Chao Liu, Lu Zhang, Hongyu Zhang

AI总结:

本文提出RAL-Bench基准,评估应用级代码生成的功能性和非功能性质量,发现功能正确性仍是瓶颈,非功能性质量也需改进。

摘要:

大型语言模型(LLMs)在代码生成方面表现出色,但大多数评估集中在片段级输出,如函数生成或仓库完成。这些设置无法全面评估应用级代码生成,目标是生成可运行的仓库,具有连贯的多文件结构、依赖支持和端到端可执行性。此外,现实软件质量不仅取决于功能性正确性,还依赖非功能性质量属性,如可维护性和安全性。本文提出了RAL-Bench,一个用于应用级代码生成的基准和评估框架。对于每个任务,RAL-Bench从高质量参考项目中推导出简洁的自然语言需求,构建黑盒系统测试以验证功能正确性和非功能性质量属性。它仅保留通过参考仓库的候选测试。在统一的评估协议下,功能正确性通过系统测试通过率衡量,而非功能性质量沿五个ISO/IEC 25010启发的维度评估,每个维度有诊断和参考标准化评分。我们评估了16个前沿LLMs,在受控的零样本设置下使用贪婪解码。结果表明,功能正确性仍是应用级代码生成的主要瓶颈,而非功能性质量也具有挑战性。在我们的评估协议下,没有模型超过45%的功能分数。这些发现表明,现有代码生成基准的强表现尚未转化为应用级仓库生成的强表现。这一结果强调了需要直接评估端到端仓库生成的评估设置,而非仅依赖片段级成功。

英文摘要:

Large language models (LLMs) have achieved strong performance on code generation. However, most prior evaluations focus on snippet-level outputs, such as function generation or repository completion. These settings do not fully evaluate application-level code generation, where the goal is to produce a runnable repository with coherent multi-file structure, dependency support, and end-to-end executability. In addition, real-world software quality depends not only on functional correctness but also on non-functional quality attributes, such as maintainability and security. In this paper, we present RAL-Bench, a benchmark and evaluation framework for application-level code generation. For each task, RAL-Bench derives a concise natural-language requirement from a high-quality reference project, constructs black-box system tests for both functional correctness and non-functional quality attributes. It also retains only the candidate tests that pass on the reference repository. Under this unified evaluation protocol, functional correctness is measured by the system test pass rate, while non-functional quality is evaluated along five ISO/IEC 25010-inspired dimensions, with per-dimension diagnostics and reference-normalized scoring.We evaluate 16 frontier LLMs under a controlled zero-shot setting with greedy decoding. The results show that functional correctness remains the primary bottleneck in application-level code generation, while non-functional quality also remains challenging. Under our evaluation protocol, no model exceeds a 45\% functional score. These findings suggest that strong performance on existing code generation benchmarks does not yet translate to strong performance on application-level repository generation. This result highlights the need for evaluation settings that directly assess end-to-end repository generation rather than relying only on snippet-level success.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑