评估大型语言模型在高等教育学生论文自动评分中的可靠性与有效性
Assessing the Reliability and Validity of Large Language Models for Automated Assessment of Student Essays in Higher Education
AI总结：
本研究评估五种LLM在高等教育论文自动评分中的可靠性，发现其与人类评分一致性低、模型内部稳定性弱，表明当前模型难以复现需学科洞察的判断，人类监督仍不可或缺。
摘要：
本研究探讨了五种先进大型语言模型（LLMs）——Claude 3.5、DeepSeek v2、Gemini 2.5、GPT-4和Mistral 24B——在真实高等教育情境下进行自动论文评分的可靠性与有效性。作为大学心理学课程一部分而撰写的共67篇意大利语学生论文，使用四项标准评分量表（相关性、连贯性、原创性、可行性）进行评估。每个模型在三次提示重复中为所有论文评分，以评估模型内部稳定性。人类与LLM之间的一致性始终较低且不显著（二次加权Kappa），而各重复之间的模型内部可靠性同样较弱（Kendall's W中位数<0.30）。出现了系统性评分分歧，包括倾向于夸大连贯性分数以及对依赖上下文的维度处理不一致。模型间一致性分析显示，在连贯性和原创性方面存在中度收敛，但在相关性和可行性方面一致性可忽略不计。尽管研究范围有限，这些发现表明，当前LLM在需要学科洞察力和情境敏感性的任务中可能难以复现人类判断。在评估开放式学术作品时，特别是在解释性领域，人类监督仍然至关重要。
英文摘要：
This study investigates the reliability and validity of five advanced Large Language Models (LLMs), Claude 3.5, DeepSeek v2, Gemini 2.5, GPT-4, and Mistral 24B, for automated essay scoring in a real world higher education context. A total of 67 Italian-language student essays, written as part of a university psychology course, were evaluated using a four-criterion rubric (Pertinence, Coherence, Originality, Feasibility). Each model scored all essays across three prompt replications to assess intra-model stability. Human-LLM agreement was consistently low and non-significant (Quadratic Weighted Kappa), and within-model reliability across replications was similarly weak (median Kendall's W < 0.30). Systematic scoring divergences emerged, including a tendency to inflate Coherence and inconsistent handling of context-dependent dimensions. Inter-model agreement analysis revealed moderate convergence for Coherence and Originality, but negligible concordance for Pertinence and Feasibility. Although limited in scope, these findings suggest that current LLMs may struggle to replicate human judgment in tasks requiring disciplinary insight and contextual sensitivity. Human oversight remains critical when evaluating open-ended academic work, particularly in interpretive domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
