arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2508.03215cs.SE
收藏

从自然语言需求生成系统模型的基准

A System Model Generation Benchmark from Natural Language Requirements

Dongming Jin, Zhi Jin, Linyu Li, Zheng Fang, Jia Li, Xiaohong Chen

AI总结:

本文提出SysMBench基准和SysMEval评估指标,用于评估大语言模型从自然语言需求生成系统模型的能力,实验表明现有LLM在此任务上表现不佳。

摘要:

系统模型是软件开发中的关键工件,提供了软件系统结构和行为方面的形式化抽象,有助于早期需求分析和架构设计。然而,由于模型描述语言的特定语法以及公共模型示例的相对稀缺,开发系统模型仍然具有挑战性。虽然大型语言模型(LLM)在生成编程语言代码方面展现出潜力,并可能有助于系统模型开发,但目前尚无基准来评估其使用特定描述语言生成系统模型的能力。我们提出了SysMBench,它包含151个人工策划的场景,涵盖广泛的流行领域和不同难度级别。每个场景主要包括一个自然语言需求描述、一个以特定模型描述语言表达的系统模型,以及一个可视化的系统模型图。需求描述作为用户输入提供给LLM,带有描述语言的系统模型用于验证生成的系统模型是否符合需求,可视化图则用于支持人工验证。我们引入了SysMEval,一种语义感知的评估指标,用于评估生成的系统模型的质量。我们使用三个传统指标和SysMEval评估了17个流行的LLM在此任务上的表现,从直接提示到三种常用的增强策略。我们的深入评估表明,LLM在SysMBench上表现不佳,最高BLEU为4%,SysMEval-F1为62%。我们发布了SysMBench及其评估框架,以促进基于LLM的系统模型生成的未来研究。

英文摘要:

System models, a critical artifact in software development, provide a formal abstraction of both the structural and behavioral aspects of software systems, which can facilitate the early requirements analysis and architecture design. However, developing system models remains challenging due to the specific syntax of model description languages and the relative scarcity of public model examples. While large language models (LLMs) have shown promise in generating code with programming languages and could potentially aid in system model development, no benchmarks currently exist for evaluating their ability to generate system models with specific description languages. We present SysMBench, which comprises 151 human-curated scenarios spanning a wide range of popular domains and varying difficulty levels. Each scenario mainly comprises a natural language requirements description, a system model expressed in a specific model description language, and a visualized system model diagram. The requirements description is fed as user input to the LLM, the system model with description language is used to verify if the generated system model conforms to the requirements, and the visualized diagram serves to support manual validation. We introduce SysMEval, a semantic-aware evaluation metric to evaluate the quality of generated system models. We evaluate 17 popular LLMs on this task with three traditional metrics and SysMEval, from directly prompting to three commonly used enhancement strategies. Our in-depth evaluation shows that LLMs perform poorly on SysMBench, with the highest BLEU of 4% and SysMEval-F1 of 62%. We release the SysMBench and its evaluation framework to enable future research on LLM-based system model generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑