arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2004.13530cs.CLcs.LG
收藏

引入一个框架来评估使用自然语言处理新创建的问题

Introducing a framework to assess newly created questions with Natural Language Processing

Politecnico di Milano(米兰理工大学);Cloud Academy Sagl

Luca Benedetto, Andrea Cappelli, Roberto Turrin, Paolo Cremonesi

摘要:

统计模型,如从项目反应理论(IRT)中得出的模型,能够对学生的特定主题进行评估,这对于几个目的(例如,学习路径定制、辍学预测)。然而,这些问题也必须进行评估,尽管可以用IRT估计已经被几个学生回答的问题的特征,但这种技术不能用于新生成的问题。在本文中,我们提出了一个框架,训练和评估模型估计的难度和歧视,新创建的多项选择题提取有意义的功能,从文本的问题和可能的选择。我们使用这个框架实现了一个模型,并在CloudAcademy提供的真实数据集上对其进行了测试,结果表明它的性能优于以前提出的模型,难度估计的RMSE降低了6.7%,区分度估计的RMSE降低了10.8%。我们还提出了一个消融研究的结果,以支持我们的功能选择,并显示不同的问题的文本的难度和歧视的特点的影响。

英文摘要:

Statistical models such as those derived from Item Response Theory (IRT) enable the assessment of students on a specific subject, which can be useful for several purposes (e.g., learning path customization, drop-out prediction). However, the questions have to be assessed as well and, although it is possible to estimate with IRT the characteristics of questions that have already been answered by several students, this technique cannot be used on newly generated questions. In this paper, we propose a framework to train and evaluate models for estimating the difficulty and discrimination of newly created Multiple Choice Questions by extracting meaningful features from the text of the question and of the possible choices. We implement one model using this framework and test it on a real-world dataset provided by CloudAcademy, showing that it outperforms previously proposed models, reducing by 6.7% the RMSE for difficulty estimation and by 10.8% the RMSE for discrimination estimation. We also present the results of an ablation study performed to support our features choice and to show the effects of different characteristics of the questions' text on difficulty and discrimination.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑