从基于特征的模型到生成式AI：构造响应评分的效度证据
From Feature-Based Models to Generative AI: Validity Evidence for Constructed Response Scoring
AI总结：
本文探讨了生成式AI在构造响应评分中的效度证据收集，对比了传统特征模型与生成式AI的差异，提出最佳实践以支持AI评分系统的使用和解释。
摘要：
大型语言模型和生成式AI的快速发展使其在高风险测试中的广泛应用成为可能。生成式AI在构造响应评分中的应用尤其有吸引力，因为它减少了传统AI评分中手工构建特征的努力，并可能超越这些方法。本文旨在突出基于特征和生成式AI在构造响应评分系统中的差异，并提出一套最佳实践，以支持使用生成式AI的评分系统中构造响应评分的使用和解释。我们比较了使用人类评分、基于特征的自然语言处理AI评分引擎和生成式AI所需的效度证据。生成式AI环境下的证据需求比基于特征的评分环境更广泛，因为缺乏透明性和生成式AI特有的问题，如一致性。来自大量独立论说作文的数据展示了不同评分系统收集效度证据的过程，并突显了在为这些评分提出效度论据时的众多复杂性和考虑因素。
英文摘要：
The rapid advancements in large language models and generative artificial intelligence (AI) capabilities are making their broad application in the high-stakes testing context more likely. Use of generative AI in the scoring of constructed responses is particularly appealing because it reduces the effort required for handcrafting features in traditional AI scoring and might even outperform those methods. The purpose of this paper is to highlight the differences in the feature-based and generative AI applications in constructed response scoring systems and propose a set of best practices for the collection of validity evidence to support the use and interpretation of constructed response scores from scoring systems using generative AI. We compare the validity evidence needed in scoring systems using human ratings, feature-based natural language processing AI scoring engines, and generative AI. The evidence needed in the generative AI context is more extensive than in the feature-based scoring context because of the lack of transparency and other concerns unique to generative AI such as consistency. Constructed response score data from a large corpus of independent argumentative essays written by 6-12th grade students demonstrate the collection of validity evidence for different types of scoring systems and highlight the numerous complexities and considerations when making a validity argument for these scores.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
