arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.04064cs.SE
收藏

端到端软件开发中基于LLM的智能体系统的基准测试与研究

Benchmarking and Studying the LLM-based Agent System in End-to-End Software Development

Peking University(北京大学)

Zhengran Zeng, Yixin Li, Rui Xie, Wei Ye, Shikun Zhang

AI总结:

本文构建E2EDevBench基准并提出混合评估框架,通过受控实证研究发现最先进LLM软件开发智能体仅能满足约50%需求,瓶颈在于需求遗漏与自我验证不足。

摘要:

基于LLM的自主智能体在端到端软件开发中的发展代表了软件工程领域的一次重大范式转变。然而,对这些系统的科学评估受到重大挑战的阻碍,包括过于简化的基准测试,以及由于混淆的实现变量而难以在不同智能体架构之间进行公平比较。为了解决这些局限性,我们首先构建了一个具有挑战性且动态维护的E2EDevBench,以模拟真实的开发场景。其次,我们提出了一个混合评估框架,该框架将基于测试用例的功能评估与细粒度、基于LLM的需求验证相结合。利用该框架,我们在统一基础上实现的三种代表性智能体架构上进行了受控实证研究,以隔离工作流设计的影响。我们的研究结果表明,最先进的智能体可以在E2EDevBench上满足约50%的需求,但其成功关键取决于任务分解与协作的架构策略。此外,我们的分析表明,主要瓶颈是需求遗漏和自我验证不足。这项工作为社区提供了一个更真实的基准测试、一个全面的评估框架,以及对软件开发智能体当前能力和核心挑战的重要见解,为未来研究指明了增强需求理解和规划的方向。

英文摘要:

The development of LLM-based autonomous agents for end-to-end software development represents a significant paradigm shift in software engineering. However, the scientific evaluation of these systems is hampered by significant challenges, including overly simplistic benchmarks and the difficulty of conducting fair comparisons between different agent architectures due to confounding implementation variables. To address these limitations, we first construct a challenging and dynamically curated E2EDevBench to simulate realistic development scenarios. Second, we propose a hybrid evaluation framework that combines test-case-based functional assessment with fine-grained, LLM-based requirement verification. Using this framework, we conduct a controlled empirical study on three representative agent architectures implemented upon a unified foundation to isolate the impact of workflow design. Our findings reveal that state-of-the-art agents can fulfill approximately 50\% of requirements on \bench{}, but their success is critically dependent on the architectural strategy for task decomposition and collaboration. Furthermore, our analysis indicates that the primary bottleneck is the omission of requirements and inadequate self-verification. This work provides the community with a more realistic benchmark, a comprehensive evaluation framework, and crucial insights into the current capabilities and core challenges of software development agents, guiding future research toward enhancing requirement comprehension and planning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑