arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.26607cs.AIcs.CYcs.SE
收藏

带有循环的人的评估异构LLM在中学数学自动能力评估中的应用

Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics

Research and Incubation Center (RAIN), Sunway College Kathmandu(研究与孵化中心(RAIN),苏尼way学院加德满都)

Jatin Bhusal, Nancy Mahatha, Aayush Acharya, Raunak Regmi

AI总结:

本文提出一种带有循环的人的评估框架,评估多个LLM在中学数学评估中的有效性,发现模型架构与指令约束的兼容性优于参数规模。

摘要:

随着以能力为基础的教育(CBE)在全球范围内获得 traction,从基于分数的评估转向定性能力映射对教育工作者来说是一个手动挑战。本文通过提出一个

英文摘要:

As Competency-Based Education (CBE) is gaining traction around the world, the shift from marks-based assessment to qualitative competency mapping is a manual challenge for educators. This paper tackles the bottleneck issue by suggesting a "Human-in-the-Loop" benchmarking framework to assess the effectiveness of multiple LLMs in automating secondary-level mathematics assessment. Based on the Grade 10 Optional Mathematics curriculum in Nepal, we created a multi-dimensional rubric for four topics and four cross-cutting competencies: Comprehension, Knowledge, Operational Fluency, and Behavior and Correlation. The multi-provider ensemble, consisted of open-weight models -- Eagle (Llama 3.1-8B) and Orion (Llama 3.3-70B) -- and proprietary frontier models Nova (Gemini 2.5 Flash) and Lyra (Gemini 3 Pro), was benchmarked against a ground truth defined by two senior mathematics faculty members (kappa_w = 0.8652). The findings show a marked "Architecture-compatibility gap". Although the Gemini-based Mixture-of-Experts (Sparse MoE) models achieved "Fair Agreement" (kappa_w ~ 0.38), the larger Orion (70B) model exhibited "No Agreement" (kappa_w = -0.0261), suggesting that architectural compliance with instruction constraints outweighs the scale of raw parameters in rubric-constrained tasks. We conclude that while LLMs are not yet suitable for autonomous certification, they provide high-value assistive support for preliminary evidence extraction within a "Human-in-the-Loop" framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑