评估大型语言模型在检测架构决策违规方面的表现
Evaluating Large Language Models for Detecting Architectural Decision Violations
AI总结：
本研究评估了大型语言模型在检测架构决策违规方面的有效性，发现其在显式决策上表现良好，但在隐式决策上存在局限，需结合专家评估。
摘要：
架构决策记录（ADRs）在维护软件架构质量中起着核心作用，然而由于项目缺乏系统化的文档和自动检测机制，许多决策违规往往被忽视。近年来，大型语言模型（LLMs）的进步为大规模自动化架构推理打开了新可能。我们研究了LLMs在开源系统中识别决策违规的有效性，通过考察其一致性、准确性和内在限制。我们的研究分析了109个GitHub仓库中的980个ADRs，采用多模型流程：一个LLM首先筛选潜在的决策违规，三个额外的LLM独立验证推理。我们评估了一致性、准确性、精确率和召回率，并用专家评估补充定量结果。模型在显式、可推导的决策上实现了显著的一致性和强准确性。对于依赖部署配置或组织知识的隐式或部署导向决策，准确性不足。因此，LLMs可以有意义地支持架构决策合规性的验证；然而，它们尚未取代人类专家对非代码聚焦的决策进行验证。
英文摘要：
Architectural Decision Records (ADRs) play a central role in maintaining software architecture quality, yet many decision violations go unnoticed because projects lack both systematic documentation and automated detection mechanisms. Recent advances in Large Language Models (LLMs) open up new possibilities for automating architectural reasoning at scale. We investigated how effectively LLMs can identify decision violations in open-source systems by examining their agreement, accuracy, and inherent limitations. Our study analyzed 980 ADRs across 109 GitHub repositories using a multi-model pipeline in which one LLM primary screens potential decision violations, and three additional LLMs independently validate the reasoning. We assessed agreement, accuracy, precision, and recall, and complemented the quantitative findings with expert evaluation. The models achieved substantial agreement and strong accuracy for explicit, code-inferable decisions. Accuracy falls short for implicit or deployment-oriented decisions that depend on deployment configuration or organizational knowledge. Therefore, LLMs can meaningfully support validation of architectural decision compliance; however, they are not yet replacing human expertise for decisions not focused on code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
