揭开自动作文评分的织锦:对准确性、公平性与泛化性的全面研究
Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability
AI总结:
本文系统评估九种自动作文评分方法,揭示提示特定模型虽准确性更高但经济状况偏差更大,而结合精心特征的传统机器学习模型在准确性与公平性上更具泛化潜力。
摘要:
自动作文评分(AES)是一项成熟的教育研究任务,它利用机器学习来评估学生撰写的作文。尽管该领域已有大量工作,但当前研究主要集中于以下两方面之一:(i)提升针对特定提示(prompt)的AES模型的预测准确性(即开发提示特定模型),这通常高度依赖来自同一目标提示的标注数据;或(ii)评估在非目标提示上开发的AES模型对预期目标提示的适用性(即在跨提示(cross-prompt)设置下开发AES模型)。鉴于机器学习中固有的偏差及其对边缘化群体的潜在影响,有必要调查当前AES方法中是否存在此类偏差,以及如果存在,它如何影响AES模型的准确性和泛化性。因此,本研究旨在揭示AES模型准确性、公平性与泛化性之间错综复杂的关系,为在现实教育中开发有效的AES模型提供实践性见解。为此,我们精心选取了九种主流的AES方法,并在一个开源数据集上使用七项指标评估其性能,该数据集包含超过25,000篇作文以及关于学生的多种人口统计信息,如性别、英语学习者身份和经济状况。通过广泛评估,我们证明:(1)提示特定模型在预测准确性方面往往优于跨提示模型;(2)与跨提示模型相比,提示特定模型经常对不同经济状况的学生表现出更大的偏差;(3)在追求泛化性的过程中,传统机器学习模型结合精心设计的特征,比复杂的神经网络模型更有潜力同时实现高准确性和高公平性。
英文摘要:
Automatic Essay Scoring (AES) is a well-established educational pursuit that employs machine learning to evaluate student-authored essays. While much effort has been made in this area, current research primarily focuses on either (i) boosting the predictive accuracy of an AES model for a specific prompt (i.e., developing prompt-specific models), which often heavily relies on the use of the labeled data from the same target prompt; or (ii) assessing the applicability of AES models developed on non-target prompts to the intended target prompt (i.e., developing the AES models in a cross-prompt setting). Given the inherent bias in machine learning and its potential impact on marginalized groups, it is imperative to investigate whether such bias exists in current AES methods and, if identified, how it intervenes with an AES model's accuracy and generalizability. Thus, our study aimed to uncover the intricate relationship between an AES model's accuracy, fairness, and generalizability, contributing practical insights for developing effective AES models in real-world education. To this end, we meticulously selected nine prominent AES methods and evaluated their performance using seven metrics on an open-sourced dataset, which contains over 25,000 essays and various demographic information about students such as gender, English language learner status, and economic status. Through extensive evaluations, we demonstrated that: (1) prompt-specific models tend to outperform their cross-prompt counterparts in terms of predictive accuracy; (2) prompt-specific models frequently exhibit a greater bias towards students of different economic statuses compared to cross-prompt models; (3) in the pursuit of generalizability, traditional machine learning models coupled with carefully engineered features hold greater potential for achieving both high accuracy and fairness than complex neural network models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。