预训练模型对新型AI辅助教育问题的教学评估评价
Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions
浏览论文内容
中文总结 AI 辅助
本研究评估预训练模型在AI辅助教育问题上的教学评估性能，发现LLM优于传统ML和BERT，文本拼接和模型重训练可提升OOD性能，为教育评估提供参考。
中文摘要 AI 辅助
AI辅助生成教育材料的激增已超出我们验证其教学质量的能力。使用Bloom分类器模型的自动评估是在大规模评估教育材料的一种有前景的方法。这些模型在分布内数据集（IID数据集）上表现出高准确性。然而，将相同的模型应用于新的分布外（OOD）数据集（如AI辅助生成的问题）可能会显示性能下降。为了识别在数据集偏移下稳健的分类器，我们评估了传统机器学习（ML）、Transformer和大语言模型在Bloom水平分类任务上的表现。我们还探索了结合NLP指标的特征工程策略，将学习目标作为输入的一部分附加，以及文本拼接以稳定OOD性能。我们的基线测试显示，TFPOS-IDF ML模型在OOD上的表现较差（Macro F1分数0.48），而BERT（0.55）和LLM（0.79）表现更好。文本拼接提高了ML和BERT模型的Macro F1分数性能（分别为0.59和0.62）。将学习目标与输入附加增加了特定数据集上的模型性能。模型重训练在模型和数据集上提供了最大的改进。总体而言，这些发现突显了使用预训练模型处理新型AI辅助教育问题的权衡，以及策略性特征增强如何帮助解决性能损失。
英文摘要
The surge in AI-assisted generation of educational materials has outpaced our capacity to validate their pedagogical quality. Automated evaluation using Bloom Classifier models is a promising approach to assess educational materials at scale. These models show high accuracy within-distribution dataset (IID Dataset). However, applying the same models to new out-of-distribution (OOD) datasets such as AI-assisted generated questions could show performance degradation. To identify robust classifiers under dataset shift, we evaluated traditional Machine Learning (ML), transformer, and Large Language models on the Bloom level classification task. We also explored feature-engineering strategies incorporating NLP metrics, appending the learning objectives as part of the input, and text splicing to stabilize OOD performance. Our baseline tests show that TFPOS-IDF ML models perform poorly on OOD (Macro F1-score 0.48) compared to BERT (0.55) and LLMs (0.79). Text splicing improved macro F1-score performance of ML and BERT models (0.59 and 0.62, respectively). Appending the learning objectives with the input increased model performance on specific dataset. Model retraining provided the largest improvement across models and datasets. Overall, these findings highlight the trade-off on the use of pre-trained models with novel AI-assisted educational questions and how strategic feature enhancements help address loss in performance.
发表机构
- Predictive Systems Inc（预测系统公司）
- Better Labs Oy
机构由 AI 辅助整理，请以论文原文为准。
