大型语言模型能否从源代码提交中提取架构设计决策？一项初步探索性研究
Can LLMs Extract Architectural Design Decisions from Source Code Commits? - A Preliminary Exploratory Study
- International Institute of Information Technology Hyderabad（海得拉巴国际信息技术学院）
- Paderborn University（帕德博恩大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究以30个开源项目的ADDs为样本，测试4种LLMs提取ADDs的效果，发现少样本提示可提升对齐度，但生成结果常缺失决策原理，为架构感知LLM系统研究提供了方向。
AI中文摘要：
背景：架构设计决策（ADDs）记录了软件系统结构与演化背后的原理，但很少被明确记录，常隐藏在源代码提交中；恢复ADDs对架构知识管理（AKM）十分重要。问题：从提交中提取ADDs颇具挑战，因其具有隐式且非结构化的特性；大型语言模型（LLMs）在理解代码和文本方面表现出强大能力，但其在该任务中的有效性尚未得到充分探索。研究：我们开展一项初步研究，采用4种LLMs（Gemini 3 Pro、DeepSeek R1、Kimi K2、Qwen3），针对来自开源项目的30个开发者编写的ADDs，使用零样本和少样本提示方法；我们用ROUGE-L、BLEU、METEOR和BERTScore对输出进行评分，且有一位作者手动审查Gemini的输出。结果：所有模型的BERT-F1均超过0.81，少样本提示提升了对齐度（Gemini的BERT-F1从0.828提升至0.847）；不过，生成的ADDs通常过长、侧重实现细节，且缺失决策背后的原理。这凸显了对架构感知LLM系统及自动化AKM的需求。
英文摘要：
Context: Architectural Design Decisions (ADDs) capture the rationale behind the structure and evolution of software systems but are rarely documented explicitly, and are often hidden inside source code commits. Recovering them is important for Architectural Knowledge Management (AKM). Problem: Extracting ADDs from commits is challenging due to their implicit and unstructured nature. Large Language Models (LLMs) have shown strong capabilities in understanding code and text, yet their effectiveness for this task remains underexplored. Study: We present a preliminary study using four LLMs (Gemini 3 Pro, DeepSeek R1, Kimi K2, Qwen3) with zeroshot and fewshot prompting on 30 developer-written ADDs from open-source projects. We score outputs with ROUGE-L, BLEU, METEOR, and BERTScore, and one author manually reviews the Gemini outputs. Results: All models reach a BERT-F1 above 0.81, and fewshot prompting improves alignment (Gemini BERT-F1: 0.828 to 0.847). However, the generated ADDs are often too long, implementation-focused, and miss the rationale behind the decision. This highlights opportunities for architecture-aware LLM systems and automated AKM.
