arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2206.08288cs.CL
收藏

平衡成本与质量:一种用于自动化简答题评分的人在环框架的探索

Balancing Cost and Quality: An Exploration of Human-in-the-loop Frameworks for Automated Short Answer Scoring

Tohoku University(东北大学);RIKEN(理化学研究所);Future Corporation(未来公司)

Hiroaki Funayama, Tasuku Sato, Yuichiroh Matsubayashi, Tomoya Mizumoto, Jun Suzuki, Kentaro Inui

摘要:

简答题评分(SAS)是对学习者写的短文进行评分的任务。近年来,基于深度学习的方法大大提高了SAS模型的性能,但在将此类模型应用于教育领域时,如何保证高质量的预测仍然是一个关键问题。为了保证高质量的预测,我们提出了第一个研究,探索使用的人在循环的框架,最大限度地减少分级成本,同时保证分级质量,允许SAS模型与人类分级共享分级任务。具体地,通过引入用于指示模型预测的可靠性的置信度估计方法,可以通过仅利用具有高可靠性的预测用于评分结果并且将具有低可靠性的预测投射给人类分级者来保证评分质量。在我们的实验中,我们使用多种置信度估计方法和多个SAS数据集调查所提出的框架的可行性。我们发现,我们的人在循环框架允许自动评分模型和人类分级,以实现目标评分质量。

英文摘要:

Short answer scoring (SAS) is the task of grading short text written by a learner. In recent years, deep-learning-based approaches have substantially improved the performance of SAS models, but how to guarantee high-quality predictions still remains a critical issue when applying such models to the education field. Towards guaranteeing high-quality predictions, we present the first study of exploring the use of human-in-the-loop framework for minimizing the grading cost while guaranteeing the grading quality by allowing a SAS model to share the grading task with a human grader. Specifically, by introducing a confidence estimation method for indicating the reliability of the model predictions, one can guarantee the scoring quality by utilizing only predictions with high reliability for the scoring results and casting predictions with low reliability to human graders. In our experiments, we investigate the feasibility of the proposed framework using multiple confidence estimation methods and multiple SAS datasets. We find that our human-in-the-loop framework allows automatic scoring models and human graders to achieve the target scoring quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑