arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13611cs.SEcs.AI
收藏

从‘是什么’到‘如何做’：利用大型语言模型弥合用户需求与软件开发之间的鸿沟

From What to How: Bridging User Requirements with Software Development Using Large Language Models

University of Science and Technology Beijing（北京科技大学）；Hong Kong University of Science and Technology（香港科学大学）；University of Science（科学大学）

Xiao He, Ru Chen, Jialun Cao

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出DesBench基准测试，评估LLMs在软件设计相关任务中的表现，揭示LLMs在代码生成、面向对象建模及测试用例设计方面的局限性。

摘要：

近年来，大型语言模型（LLMs）被广泛用于提高开发效率，导致了大量评估其性能的基准测试。然而，这些基准测试主要集中在实现上，忽视了同样关键的软件设计方面。这一空白引发了两个关键问题：（1）LLMs能否处理软件设计？（2）LLMs能否根据特定设计编写代码？为探讨这些问题，本文提出了DesBench，一个设计感知的基准测试，用于评估LLMs在三个软件设计相关任务上的表现：设计感知的代码生成、面向对象建模以及接受测试用例的设计。DesBench包含30个手动创建的Java项目，包括需求文档、设计模型、实现和接受测试，总计30个设计模型、194个Java类和737个测试用例。我们使用DesBench评估了七种最先进的LLMs，包括三个DeepSeek R1、两个Qwen2.5和两个GPT模型。结果表明，LLMs在软件设计的复杂性方面仍然面临显著挑战：（1）在代码生成方面，当仅提供高层次或没有设计时，LLMs难以生成正确的实现。（2）在面向对象建模方面，虽然LLMs可以准确识别对象和类，但定义操作和类间关系存在困难。（3）由LLMs根据功能需求生成的接受测试用例在代码覆盖率质量上与人类编写的一致。我们的研究突显了LLMs在管理软件设计方面的当前局限，并呼吁进一步研究适用于基于LLM开发的新设计方法和语言。

英文摘要：

Recently, large language models (LLMs) are extensively utilized to enhance development efficiency, leading to numerous benchmarks for evaluating their performance. However, these benchmarks predominantly focus on implementation, overlooking the equally critical aspect of software design. This gap raises two pivotal questions: (1) Can LLMs handle software design? (2) Can LLMs write code following the specific designs? To investigate these questions, this paper proposes DesBench, a design-aware benchmark for evaluating LLMs on three software design-related tasks: design-aware code generation, object-oriented modeling, and the design of acceptance test cases. DesBench comprises 30 manually crafted Java projects that include requirement documents, design models, implementations, and acceptance tests, amounting to a total of 30 design models, 194 Java classes, and 737 test cases. We evaluated seven state-of-the-art LLMs, including three DeepSeek R1, two Qwen2.5, and two GPT models, using DesBench. The results reveal that LLMs remain significantly challenged by the intricacies of software design: (1) For code generation, LLMs struggle to produce correct implementations when provided with only high-level or no designs. (2) In object-oriented modeling, while LLMs can accurately identify objects and classes, they face challenges in defining operations and inter-class relationships. (3) Acceptance test cases generated by LLMs from functional requirements achieve code coverage quality comparable to those written by humans. Our research highlights the current limitations of LLMs in managing software design and calls for further investigation into new design methodologies and languages suitable for LLM-based development.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑