arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.10899cs.CYcs.AIcs.CL
收藏

利用大语言模型作答者进行题目评估:一项心理测量学分析

Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis

Berkeley School of Education, University of California, Berkeley(加州大学伯克利分校教育学院);Electrical Engineering and Computer Science, University of California, Berkeley(加州大学伯克利分校电气工程与计算机科学学院)

Yunting Liu, Shreya Bhandari, Zachary A. Pardos

AI总结:

本研究探索利用多种大语言模型及集成、增强方法模拟人类作答进行题目标定,发现LLM集成可拟合人类能力分布,重采样增强能提升题目参数标定的相关性,为降低教育测量中题目标定的成本与耗时提供新路径。

摘要:

有效的教育测量高度依赖精心设计的题库(即具备合适的心理测量学属性)的构建。然而,题目标定耗时且成本高昂,需要足够数量的作答者参与答题过程。本研究探索使用六种不同的大语言模型(LLM)——GPT-3.5、GPT-4、Llama 2、Llama 3、Gemini-Pro和Cohere Command R Plus,以及通过采样方法组合它们的不同形式,生成具备与人类作答相似心理测量学属性的回答。结果显示,部分LLM在大学代数科目上的能力与大学生相当甚至更高。由于能力分布较窄,单一LLM无法模拟人类作答者,但LLM集成模型能更好地拟合大学生的能力分布。由LLM作答者标定的题目参数与人类标定的参数具有高相关性(例如GPT-3.5的相关系数大于0.8),且与人类子集的参数高度接近(例如斯皮尔曼相关系数差异为0.02)。研究还评估了多种增强策略的相对表现,其中重采样方法效果最佳,将斯皮尔曼相关系数从仅使用人类数据的0.89提升至增强后人类数据的0.93。

英文摘要:

Effective educational measurement relies heavily on the curation of well-designed item pools (i.e., possessing the right psychometric properties). However, item calibration is time-consuming and costly, requiring a sufficient number of respondents for the response process. We explore using six different LLMs (GPT-3.5, GPT-4, Llama 2, Llama 3, Gemini-Pro, and Cohere Command R Plus) and various combinations of them using sampling methods to produce responses with psychometric properties similar to human answers. Results show that some LLMs have comparable or higher proficiency in College Algebra than college students. No single LLM mimics human respondents due to narrow proficiency distributions, but an ensemble of LLMs can better resemble college students' ability distribution. The item parameters calibrated by LLM-Respondents have high correlations (e.g. > 0.8 for GPT-3.5) compared to their human calibrated counterparts, and closely resemble the parameters of the human subset (e.g. 0.02 Spearman correlation difference). Several augmentation strategies are evaluated for their relative performance, with resampling methods proving most effective, enhancing the Spearman correlation from 0.89 (human only) to 0.93 (augmented human).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑