ZPD-SCA：揭示大语言模型在评估学生认知能力方面的盲区
ZPD-SCA: Unveiling the Blind Spots of LLMs in Assessing Students' Cognitive Abilities
AI总结：
本文提出ZPD-SCA基准，由60名特级教师标注中文阅读理解难度，用于评估大语言模型判断阅读材料与学生认知能力匹配程度的能力，实验揭示LLMs在该任务上存在零样本表现差、方向性偏差和体裁差异大等问题，为改进认知对齐的教育应用提供基础。
摘要：
大语言模型（LLMs）已在教育应用中展现出潜力，然而它们准确评估阅读材料与学生发展阶段之间认知匹配程度的能力仍未得到充分探索。鉴于最近发展区（ZPD）这一基础性教育原则强调需要将学习资源与学生的认知能力（SCA）相匹配，这一空白尤为关键。尽管这种匹配十分重要，但目前明显缺乏系统的研究来考察大语言模型在不同学生年龄组中评估阅读理解难度的能力，尤其是在中文教育背景下。为填补这一空白，我们引入了ZPD-SCA，这是一个专门设计用于评估阶段级中文阅读理解难度的新型基准。该基准由60名特级教师进行标注，这一群体代表了全国所有在职教师中前0.15%的顶尖水平。实验结果表明，大语言模型在零样本学习场景中表现不佳，Qwen-max和GLM的表现甚至低于随机猜测的概率。当提供上下文示例时，大语言模型的性能大幅提升，部分模型的准确率几乎达到其零样本基线的两倍。这些结果揭示了大语言模型具备评估阅读难度的新兴能力，同时也暴露了其当前训练在教育对齐判断方面的局限性。值得注意的是，即使是表现最好的模型也显示出系统性的方向偏差，表明在将材料难度与SCA准确对齐方面存在困难。此外，模型在不同体裁上的表现存在显著差异，凸显了该任务的复杂性。我们期望ZPD-SCA能够为评估和改进大语言模型在认知对齐的教育应用中的表现提供基础。
英文摘要：
Large language models (LLMs) have demonstrated potential in educational applications, yet their capacity to accurately assess the cognitive alignment of reading materials with students' developmental stages remains insufficiently explored. This gap is particularly critical given the foundational educational principle of the Zone of Proximal Development (ZPD), which emphasizes the need to match learning resources with Students' Cognitive Abilities (SCA). Despite the importance of this alignment, there is a notable absence of comprehensive studies investigating LLMs' ability to evaluate reading comprehension difficulty across different student age groups, especially in the context of Chinese language education. To fill this gap, we introduce ZPD-SCA, a novel benchmark specifically designed to assess stage-level Chinese reading comprehension difficulty. The benchmark is annotated by 60 Special Grade teachers, a group that represents the top 0.15% of all in-service teachers nationwide. Experimental results reveal that LLMs perform poorly in zero-shot learning scenarios, with Qwen-max and GLM even falling below the probability of random guessing. When provided with in-context examples, LLMs performance improves substantially, with some models achieving nearly double the accuracy of their zero-shot baselines. These results reveal that LLMs possess emerging abilities to assess reading difficulty, while also exposing limitations in their current training for educationally aligned judgment. Notably, even the best-performing models display systematic directional biases, suggesting difficulties in accurately aligning material difficulty with SCA. Furthermore, significant variations in model performance across different genres underscore the complexity of task. We envision that ZPD-SCA can provide a foundation for evaluating and improving LLMs in cognitively aligned educational applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
