arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.29520cs.SEcs.AIcs.DB
收藏

SAKE:大型语言模型的软件架构知识评估基准

SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

University of Southern Denmark(丹麦南方大学)

Tiziano Santilli, Francesco Daghero, Mayhar Tourchi Moghaddam

AI总结:

提出SAKE基准,包含2154道专家策划的多选题,评估LLM在八个架构类别和四种上下文长度下的架构知识,揭示专业实践中的能力差距。

摘要:

大型语言模型(LLM)越来越多地被用作整个软件开发生命周期中的助手,但它们对软件架构进行推理的能力在很大程度上仍未得到衡量。架构决策依赖于质量属性权衡、设计模式和系统级约束,而针对语法或算法任务的基准测试并未涉及这些方面。我们引入了SAKE(软件架构知识评估),这是一个标准化且可重复的基准,用于评估LLM中的软件架构知识。SAKE包含2154道专家策划的多选题,每道题有四个选项,分布在八个架构类别和四种上下文长度级别中。我们在零样本和五样本设置下评估了11个专有和开源模型。总体准确率较高,但不同类别的表现差异显著,揭示了专业实践核心领域的能力差距。SAKE、其评估脚本和所有结果均作为开源发布,以便社区为跟踪LLM中的架构推理建立基线。

英文摘要:

Large Language Models (LLMs) are increasingly used as assistants across the software development lifecycle, yet their ability to reason about software architecture remains largely unmeasured. Architectural decision-making depends on quality attribute trade-offs, design patterns, and system-level constraints, none of which are exercised by benchmarks that target syntactic or algorithmic tasks. We introduce SAKE (Software Architectural Knowledge Evaluation), a standardized and reproducible benchmark for assessing software architectural knowledge in LLMs. SAKE comprises 2154 expert-curated multiple-choice questions, each with four options, stratified across eight architectural categories and four context-length levels. We evaluate 11 proprietary and open-weight models in zero-shot and five-shot settings. Overall accuracy is high, but performance varies markedly across categories, revealing competency gaps in areas central to professional practice. SAKE, its evaluation scripts, and all results are released as open source to give the community a baseline for tracking architectural reasoning in LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑