评估大语言模型在前端软件架构知识方面的能力
Assessing LLMs for Front-end Software Architecture Knowledge
AI总结：
本研究基于布鲁姆分类法构建评估框架，考察大语言模型对 iOS VIPER 架构的理解、复现与生成能力，发现其在评价和创造等高阶任务中表现优异，但在需要精确检索架构细节的低阶任务中存在不足，并提出一种面向软件架构能力评估的基准格式。
摘要：
大语言模型（LLMs）在自动化软件开发任务方面已展现出显著前景，但它们在软件设计任务方面的能力在很大程度上仍不明确。本研究考察了大语言模型在理解、复现和生成复杂 VIPER 架构（一种 iOS 应用设计模式）中结构的能力。我们利用布鲁姆分类法开发了一个综合评估框架，以评估该大语言模型在记忆、理解、应用、分析、评价和创造等不同认知领域的表现。使用 ChatGPT 4 Turbo 2024-04-09 进行的实验结果表明，该大语言模型在评价和创造等高阶任务中表现优异，但在需要精确检索架构细节的低阶任务中面临挑战。这些发现既凸显了大语言模型降低开发成本的潜力，也揭示了其在真实软件设计场景中有效应用的障碍。本研究提出了一种用于评估大语言模型在软件架构方面能力的基准格式，旨在为更稳健、更易用的 AI 驱动开发工具做出贡献。
英文摘要：
Large Language Models (LLMs) have demonstrated significant promise in automating software development tasks, yet their capabilities with respect to software design tasks remains largely unclear. This study investigates the capabilities of an LLM in understanding, reproducing, and generating structures within the complex VIPER architecture, a design pattern for iOS applications. We leverage Bloom's taxonomy to develop a comprehensive evaluation framework to assess the LLM's performance across different cognitive domains such as remembering, understanding, applying, analyzing, evaluating, and creating. Experimental results, using ChatGPT 4 Turbo 2024-04-09, reveal that the LLM excelled in higher-order tasks like evaluating and creating, but faced challenges with lower-order tasks requiring precise retrieval of architectural details. These findings highlight both the potential of LLMs to reduce development costs and the barriers to their effective application in real-world software design scenarios. This study proposes a benchmark format for assessing LLM capabilities in software architecture, aiming to contribute toward more robust and accessible AI-driven development tools.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
