arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.04009cs.SE
收藏

对软件架构图理解的VLMs基准测试与评估

Benchmarking and Evaluating VLMs for Software Architecture Diagram Understanding

King’s College London(伦敦国王学院);British Telecom(英国电信)

Shuyin Ouyang, Jie M. Zhang, Jingzhi Gong, Gunel Jahangirova, Mohammad Reza Mousavi, Jack Johns, Beum Seuk Lee, Adam Ziolkowski, Botond Virginas, Joost Noppen

AI总结:

本文提出SADU基准,评估VLMs在理解软件架构图作为结构化软件工程制品上的能力,揭示当前模型在图表推理和视觉关系定位方面的不足。

摘要:

软件架构图是沟通系统结构、行为和数据组织的重要设计 artifact,在软件开发生命周期中至关重要。尽管大语言模型的最新进展显著提升了代码中心的软件工程任务,如代码生成、测试和维护,但现代视觉语言模型(VLMs)理解软件架构图的能力仍待探索。为填补这一空白,我们提出了SADU,一个用于软件架构图理解的基准,评估VLMs在架构图作为结构化软件工程制品上的能力,而非通用图像。SADU包含154精心挑选的图表,涵盖行为、结构和ER图表,配以结构化注释和2,431个问题-答案任务,涵盖计数和检索推理。我们评估了11种最先进的VLMs,来自Gemini、Claude、GPT和Qwen家族。结果表明,当前模型在理解软件架构图上仍面临挑战:表现最好的模型Gemini-3-Flash-Preview仅达到70.18%的准确率,而GPT-4O-Mini仅达到17.77%的准确率。结果进一步揭示了图表推理和视觉关系定位的不足,突显了当前VLMs与设计阶段软件工程需求之间的差距。SADU为未来研究图感知AI系统和更忠实的AI辅助软件工程工作流程提供了基础。

英文摘要:

Software architecture diagrams are important design artifacts for communicating system structure, behavior, and data organization throughout the software development lifecycle. Although recent progress in large language models has substantially advanced code-centric software engineering tasks such as code generation, testing, and maintenance, the ability of modern vision-language models (VLMs) to understand software architecture diagrams remains underexplored. To address this gap, we present SADU, a benchmark for Software Architecture Diagram Understanding that evaluates VLMs on architecture diagrams as structured software engineering artifacts rather than generic images. SADU contains 154 carefully curated diagrams spanning behavioral, structural, and ER diagrams, paired with structured annotations and 2,431 question-answer tasks covering counting and retrieval reasoning. We evaluate 11 state-of-the-art VLMs from the Gemini, Claude, GPT, and Qwen families. Our results show that software architecture diagram understanding remains challenging for current models: the best-performing model gemini-3-flash-preview achieves only 70.18\% accuracy, while gpt-4o-mini only achieves 17.77\% accuracy. The results further reveal the weaknesses in diagram reasoning and visual relation grounding, highlighting a gap between current VLMs and the needs of design-stage software engineering. SADU provides a foundation for future research on diagram-aware AI systems and more faithful AI-assisted software engineering workflows.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑