arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.01730cs.CLcs.HC
收藏

测验设计任务:帮助教师通过自动问题生成创建测验

Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation

Salesforce AI Research(赛富时人工智能研究院)

Philippe Laban, Chien-Sheng Wu, Lidiya Murakhovs'ka, Wenhao Liu, Caiming Xiong

摘要:

问题生成(QGen)模型通常使用基于n-gram重叠的标准化NLG指标进行评估。在本文中,我们衡量这些指标的改进是否转化为实际环境中的收益,重点是帮助教师自动生成阅读理解测验的用例。在我们的研究中,教师建立一个测验收到的问题建议,他们可以接受或拒绝的理由。尽管我们发现QGen的最新进展导致问题接受率显着提高,但仍有很大的改进空间,最好的模型只有68.4%的问题被参与我们研究的十位教师接受。然后,我们利用我们收集的注释来分析标准的NLG指标,发现模型性能已经达到了预期的上限,这表明需要新的自动指标来指导QGen研究。

英文摘要:

Question generation (QGen) models are often evaluated with standardized NLG metrics that are based on n-gram overlap. In this paper, we measure whether these metric improvements translate to gains in a practical setting, focusing on the use case of helping teachers automate the generation of reading comprehension quizzes. In our study, teachers building a quiz receive question suggestions, which they can either accept or refuse with a reason. Even though we find that recent progress in QGen leads to a significant increase in question acceptance rates, there is still large room for improvement, with the best model having only 68.4% of its questions accepted by the ten teachers who participated in our study. We then leverage the annotations we collected to analyze standard NLG metrics and find that model performance has reached projected upper-bounds, suggesting new automatic metrics are needed to guide QGen research forward.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑