使用大语言模型对Linux/bash考试进行自动评分：一种四层认知分类方法
Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach
AI总结：
本研究评估四种前沿大语言模型（GPT、Claude Opus、Gemini、GLM）在Linux/bash命令评分中的表现，采用四层认知分类法，发现Gemini 3.0 Pro配合评分标准提示达到最高人机一致性（ICC=0.888），且问题复杂度是评分难度的可靠预测因子。
摘要：
在计算教育中，命令行考试的可扩展且可靠的评分仍然是一个挑战，因为入学人数增加使得人工评分困难，而基于规则的自动评分器无法处理部分分数、等效解决方案或语法变化。本文评估了四种前沿大语言模型（GPT、Claude Opus、Gemini和GLM）在评分简短Linux/bash命令回答时是否能接近专家判断。研究采用四层认知分类法，结合认知复杂度和操作影响，范围从信息检索（L1）和基本文件操作（L2）到结构操作（L3）和高级系统管理（L4）。模型使用两种提示变体（最小基线和增强评分标准版本）进行测试，测试数据为来自二年级计算机工程学生的1200个真实回答，由三位专家讲师独立评分。Gemini 3.0 Pro配合评分标准引导提示达到了最高的人机一致性（ICC(3,1) = 0.888，MAE = 0.10，Bland-Altman偏差 = -0.014）。随着分类层次增加，一致性持续下降，在较高层次上差异最大。在所有模型中，评分标准质量的影响大于模型提供商选择，结构化提示持续提高一致性。这些结果表明，问题复杂度是LLMs准确评分难度的可靠预测因子，并建立了一个基于分类法的原则性框架，用于确定哪些问题适合AI辅助评分，哪些需要人工审核，同时提供了可迁移的评估协议和提示模板。
英文摘要：
Scalable and reliable grading of command-line examinations remains a challenge in computing education, where rising enrolments make manual marking difficult and rule-based autograders cannot handle partial credit, equivalent solutions, or syntactic variation. This paper evaluates whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The study adopts a four-level cognitive taxonomy that combines cognitive complexity and operational impact, ranging from information retrieval (L1) and basic file manipulation (L2) to structural operations (L3) and advanced system management (L4). The models were tested with two prompt variants, a minimal baseline and a rubric-enhanced version, on 1200 real responses from second-year Computer Engineering students independently graded by three expert instructors. Gemini~3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014). Agreement declined consistently as taxonomy level increased, with the largest discrepancies at higher levels. Across all models, rubric quality had a larger effect than provider choice, with structured prompts consistently improving agreement. These results show that question complexity is a reliable predictor of the difficulty LLMs face in grading accurately, and they establish a principled, taxonomy-based framework for determining which questions are suitable for AI-assisted grading and which require human review, while also providing a transferable evaluation protocol and prompt templates.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
