在Scrum认证风格问题上比较大型语言模型：准确性、稳定性和错误模式
Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns
AI总结：
评估GPT-5 mini、Gemini 3 Flash和DeepSeek Chat 3.2在993个Scrum认证问题上的表现，发现模型间差异显著，Gemini 3 Flash准确率最高，错误模式具有系统性。
摘要：
大型语言模型（LLMs）越来越多地用于考试和认证风格的问题回答任务，在这些任务中，可以系统地评估它们检索、解释和应用领域特定知识的能力。在软件工程中，当问题依赖于严格遵守规范性定义、角色、工件和规则时，这种设置尤其相关。本文评估了三个当代LLM——\textit{GPT-5 mini}、\textit{Gemini 3 Flash}和\textit{DeepSeek Chat 3.2}——在回答993个与Professional Scrum Master I（PSM I）评估格式一致的Scrum认证风格问题上的表现。我们在三种提示策略（\textit{零样本}、\textit{思维链}和\textit{源基础}）下评估了模型，并通过重复执行来评估模型内稳定性。我们还分析了跨Scrum主题和问题格式的性能，并辅以对错误答案中反复出现的错误模式的定性分析。结果揭示了模型之间的明显差异，Gemini 3 Flash实现了最高准确率，其次是GPT-5 mini和DeepSeek Chat 3.2，而所有条件下的模型内变异性仍然很低。按问题格式，模型在单选多项选择题上达到了最高准确率，而多选和判断题更容易出错。按主题，在规范明确的领域（如工件、经验主义和产品价值）中性能更一致，但在Scrum价值观、自管理团队和利益相关者与客户方面则更脆弱。定性分析表明，错误是系统性的而非随机性的，涉及过度泛化、限制性措辞、复合干扰项以及常见市场解释与严格Scrum定义之间的冲突。
英文摘要：
Large Language Models (LLMs) are increasingly used in exam- and certification-style question answering tasks, where their ability to retrieve, interpret, and apply domain-specific knowledge can be systematically assessed. In Software Engineering, such settings are particularly relevant when questions depend on strict adherence to normative definitions, roles, artifacts, and rules. This paper evaluates the performance of three contemporary LLMs, \textit{GPT-5 mini}, \textit{Gemini 3 Flash}, and \textit{DeepSeek Chat 3.2}, in answering 993 Scrum certification-style questions aligned with the Professional Scrum Master I (PSM I) assessment format. We evaluated the models under three prompting strategies (\textit{zero-shot}, \textit{chain-of-thought}, and \textit{source-grounded}), with repeated executions to assess intra-model stability. We also analyzed performance across Scrum topics and question formats, complemented by a qualitative analysis of recurring error patterns in incorrect answers. Results revealed clear differences among models, with Gemini 3 Flash achieving the highest accuracy, followed by GPT-5 mini and DeepSeek Chat 3.2, while intra-model variability remained low across all conditions. By question format, the models achieved the highest accuracy on single-answer multiple-choice items, whereas multi-select and True/False questions were more error-prone. By topic, performance was more consistent in normatively explicit areas such as Artifacts, Empiricism, and Product Value, but more fragile in Scrum Values, Self-Managing Teams, and Stakeholders \& Customers. The qualitative analysis showed that errors were systematic rather than random, involving overgeneralization, restrictive wording, compound distractors, and conflicts between common market interpretations and strict Scrum definitions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
