arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.03512cs.CYcs.AI
收藏

AI工具在分类数学任务认知需求中的基准性能

Baseline Performance of AI Tools in Classifying Cognitive Demand of Mathematical Tasks

Learning Research and Development Center, University of Pittsburgh(匹兹堡大学学习研究与发展中心);Institute for Learning, University of Pittsburgh(匹兹堡大学学习研究所)

Danielle S. Fox, Brenda L. Robles, Elizabeth DiPietro Brovey, Christian D. Schunn

AI总结:

研究评估了AI工具在分类数学任务认知需求的准确性,发现通用工具与教育专用工具均存在显著误差,需改进提示工程和工具开发。

摘要:

教师面临日益增长的时间压力,特别是在将数学课程适应于满足个别学生需求的同时保持认知严谨性方面。本研究评估了AI工具能否准确分类数学任务的认知需求,这对于创建或调整支持学生学习的任务至关重要。我们测试了十一款AI工具:六款通用型(ChatGPT、Claude、DeepSeek、Gemini、Grok、Perplexity)和五款教育专用型(Brisk、Coteach AI、Khanmigo、Magic School、School.AI),以评估它们在使用基于研究的框架对数学任务进行四类认知需求分类的能力。目标是近似教师在使用简单提示时的表现。平均而言,AI工具仅在63%的情况下准确分类认知需求。教育专用工具的准确性并不比通用工具更高,且没有工具超过83%的准确率。所有工具在认知需求极端任务(记忆和做数学)上均表现不佳,显示出系统性偏见,倾向于中等类别水平(带有/无连接的程序)。工具常常给出听起来合理且可能说服新手教师的解释。对AI工具在认知需求广度(高 vs. 低)上的误分类分析揭示,工具一致倾向于表面文本特征而非潜在的认知过程。此外,AI工具在推理任务因素以区分高 vs. 低认知需求方面存在弱点。错误并非源于忽略相关维度,而是源于对多个任务方面错误推理。这些发现对AI在教师规划工作流程中的整合具有影响,并突显了改进提示工程和教育应用工具开发的必要性。

英文摘要:

Teachers face increasing demands on their time, particularly in adapting mathematics curricula to meet individual student needs while maintaining cognitive rigor. This study evaluates whether AI tools can accurately classify the cognitive demand of mathematical tasks, which is important for creating or adapting tasks that support student learning. We tested eleven AI tools: six general-purpose (ChatGPT, Claude, DeepSeek, Gemini, Grok, Perplexity) and five education-specific (Brisk, Coteach AI, Khanmigo, Magic School, School$.$AI), on their ability to categorize mathematics tasks across four levels of cognitive demand using a research-based framework. The goal was to approximate the performance teachers will achieve with straightforward prompts. On average, AI tools accurately classified cognitive demand in only 63% of cases. Education-specific tools were not more accurate than general-purpose tools, and no tool exceeded 83% accuracy. All tools struggled with tasks at the extremes of cognitive demand (Memorization and Doing Mathematics), exhibiting a systematic bias toward middle-category levels (Procedures with/without Connections). The tools often gave plausible-sounding explanations likely to be persuasive to novice teachers. Error analysis of AI tools' misclassification of the broad level of cognitive demand (high vs. low) revealed that tools consistently overweighted surface textual features over underlying cognitive processes. Further, AI tools showed weaknesses in reasoning about factors that make tasks higher vs. lower cognitive demand. Errors stemmed not from ignoring relevant dimensions, but from incorrectly reasoning about multiple task aspects. These findings carry implications for AI integration into teacher planning workflows and highlight the need for improved prompt engineering and tool development for educational applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑