使用生成式人工智能应用进行构作反应评分的效度论证
Validity Arguments For Constructed Response Scoring Using Generative Artificial Intelligence Applications
AI总结：
本文比较人工评分、基于特征的NLP评分与生成式AI评分，提出为生成式AI构作反应评分收集更广泛效度证据的最佳实践及多AI贡献式评分思路。
摘要：
大型语言模型和生成式人工智能（AI）能力的快速发展，正使其在高利害测试情境中的广泛应用变得更有可能。在构作反应评分中使用生成式AI尤其具有吸引力，因为它减少了传统AI评分中手工构建特征所需的工作量，甚至可能优于那些方法。本文旨在强调构作反应评分系统中基于特征的应用与生成式AI应用之间的差异，并提出一套收集效度证据的最佳实践，以支持对使用生成式AI的评分系统所给出的构作反应分数的使用和解释。我们比较了使用人工评分、基于特征的自然语言处理AI评分引擎以及生成式AI的评分系统所需的效度证据。由于缺乏透明度以及生成式AI特有的其他问题（如一致性），生成式AI情境中所需的证据比基于特征的NLP评分情境更为广泛。来自标准化测试的构作反应分数数据展示了针对不同类型评分系统的效度证据收集，并凸显了在为这些分数构建效度论证时的众多复杂性和考虑因素。此外，我们还讨论了对AI分数的评估如何可能纳入对一种贡献式评分方法的考量，该方法结合多个AI分数（来自不同来源），在没有人工评分的情况下覆盖更多构念。
英文摘要：
The rapid advancements in large language models and generative artificial intelligence (AI) capabilities are making their broad application in the high-stakes testing context more likely. Use of generative AI in the scoring of constructed responses is particularly appealing because it reduces the effort required for handcrafting features in traditional AI scoring and might even outperform those methods. The purpose of this paper is to highlight the differences in the feature-based and generative AI applications in constructed response scoring systems and propose a set of best practices for the collection of validity evidence to support the use and interpretation of constructed response scores from scoring systems using generative AI. We compare the validity evidence needed in scoring systems using human ratings, feature-based natural language processing AI scoring engines, and generative AI. The evidence needed in the generative AI context is more extensive than in the feature-based NLP scoring context because of the lack of transparency and other concerns unique to generative AI such as consistency. Constructed response score data from standardized tests demonstrate the collection of validity evidence for different types of scoring systems and highlights the numerous complexities and considerations when making a validity argument for these scores. In addition, we discuss how the evaluation of AI scores might include a consideration of how a contributory scoring approach combining multiple AI scores (from different sources) will cover more of the construct in the absence of human ratings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
