arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.18536stat.APcs.SE
收藏

题库质量保证分析(AQuAP):AI驱动评估系统中题库健康的监控与维护

Analytics for Quality Assurance for Item Pools (AQuAP): Monitoring and Maintaining Item Bank Health in AI-Driven Assessment Systems

Duolingo(多邻国)

Alina A. von Davier, Xiaowan Zhang, Yigal Attali, Yena Park, Jacqueline Church, Andrew Runge, Geoff T. LaFlair, Alexander Tsigler

AI总结:

提出AQuAP仪表盘环境,通过有效题库规模等指标监控题库质量,支持大规模自动与人工结合的试题开发,确保高利害测试的题库健康。

摘要:

教育评估的大规模数字化使得题库的持续监督既必要又复杂。本文提出了题库质量保证分析(AQuAP),一个用于监控试题质量和题库健康的仪表盘环境。AQuAP支持高利害测试中大规模试题生成程序的操作实施,这些程序包含在试题工厂(一个自动化和人工支持的测试开发框架)中。本文描述了AQuAP与试题开发过程的关系,概述了题库质量保证的更广泛度量框架,并强调了有效题库规模(EBS)作为题库活力的核心指标。EBS量化了在内容重复发生之前可以构建的独立测试会话数量,当与曝光度和使用度量结合时,它提供了对题库安全性、多样性和效率的洞察。我们进一步引入了题库健康度量,如最大曝光度、最大条件曝光度、调整后的有效题库规模和极少施测比例,所有这些都扩展了试题利用情况的图景。AQuAP展示了操作分析如何将心理测量概念转化为高容量、AI驱动的测试程序的质量保证工具。本文以多邻国英语测试(DET)流程为例进行说明。

英文摘要:

The large-scale digitization of educational assessment has made the continuous oversight of item banks both essential and complex. This paper presents Analytics for Quality Assurance for Item Pools (AQuAP), a dashboard environment for monitoring item quality and item bank health. AQuAP supports the operational implementation of the large scale item generation procedures for high-stakes tests as included in the Item Factory, a framework for automated and human-supported test development. The paper describes AQuAP in relationship with the process of item development, outlines the broader metric framework for item-pool quality assurance, and highlights the Effective Bank Size (EBS) as one central indicator of pool vitality. EBS quantifies how many independent test sessions can be constructed before content repetition occurs and, when coupled with exposure and usage metrics, provides insight into item bank security, diversity, and efficiency. We further introduce bank-health metrics, such as maximum exposure, maximum conditional exposure, adjusted effective bank size, and the rarely-administered fraction, all of which extend this picture of item utilization. AQuAP illustrates how operational analytics can translate psychometric concepts into quality assurance tools for high-volume, AI-enabled testing programs. This work is illustrated with the Duolingo English Test (DET) processes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑