Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis
Berkeley School of Education, University of California, Berkeley(加州大学伯克利分校教育学院);Electrical Engineering and Computer Science, University of California, Berkeley(加州大学伯克利分校电气工程与计算机科学学院)
有效的教育测量高度依赖精心设计的题库(即具备合适的心理测量学属性)的构建。然而,题目标定耗时且成本高昂,需要足够数量的作答者参与答题过程。本研究探索使用六种不同的大语言模型(LLM)——GPT-3.5、GPT-4、Llama 2、Llama 3、Gemini-Pro和Cohere Command R Plus,以及通过采样方法组合它们的不同形式,生成具备与人类作答相似心理测量学属性的回答。结果显示,部分LLM在大学代数科目上的能力与大学生相当甚至更高。由于能力分布较窄,单一LLM无法模拟人类作答者,但LLM集成模型能更好地拟合大学生的能力分布。由LLM作答者标定的题目参数与人类标定的参数具有高相关性(例如GPT-3.5的相关系数大于0.8),且与人类子集的参数高度接近(例如斯皮尔曼相关系数差异为0.02)。研究还评估了多种增强策略的相对表现,其中重采样方法效果最佳,将斯皮尔曼相关系数从仅使用人类数据的0.89提升至增强后人类数据的0.93。
英文摘要:
Effective educational measurement relies heavily on the curation of well-designed item pools (i.e., possessing the right psychometric properties). However, item calibration is time-consuming and costly, requiring a sufficient number of respondents for the response process. We explore using six different LLMs (GPT-3.5, GPT-4, Llama 2, Llama 3, Gemini-Pro, and Cohere Command R Plus) and various combinations of them using sampling methods to produce responses with psychometric properties similar to human answers. Results show that some LLMs have comparable or higher proficiency in College Algebra than college students. No single LLM mimics human respondents due to narrow proficiency distributions, but an ensemble of LLMs can better resemble college students' ability distribution. The item parameters calibrated by LLM-Respondents have high correlations (e.g. > 0.8 for GPT-3.5) compared to their human calibrated counterparts, and closely resemble the parameters of the human subset (e.g. 0.02 Spearman correlation difference). Several augmentation strategies are evaluated for their relative performance, with resampling methods proving most effective, enhancing the Spearman correlation from 0.89 (human only) to 0.93 (augmented human).
Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators
Graduate School of Data Science, Seoul National University(首尔国立大学数据科学研究生院);Department of Communication, Seoul National University(首尔国立大学通信系);Interdisciplinary Program in Artificial Intelligence, Seoul National University(首尔国立大学人工智能跨学科项目)
Neural network embeddings recover value dimensions from psychometric survey items on par with human data
Barcelona Supercomputing Center(巴塞罗那超级计算中心);Jam Technologies GmbH;University of Mannheim(曼海姆大学);GESIS - Leibniz Institute for the Social Sciences(莱布尼茨社会科学研究所)
The Ultimate Tutorial for AI-driven Scale Development in Generative Psychometrics: Releasing AIGENIE from its Bottle
Department of Psychology, University of Virginia(弗吉尼亚大学心理学系);Department of Psychology, Pontificia Universidad Madre y Maestra(宗座圣母大学心理学系);Department of Psychology, Vanderbilt University(范德比尔特大学心理学系)
Human Psychometric Questionnaires Mischaracterize LLM Behavior
Graduate School of Data Science, Seoul National University(首尔国立大学数据科学研究生院);Department of Communication, Interdisciplinary Program in Artificial Intelligence, Seoul National University(首尔国立大学通信系人工智能交叉学科项目)
Microsoft Research Asia(微软亚洲研究院);Beijing Normal University(北京师范大学);Universitat Politècnica de València(瓦伦西亚理工大学);ValgrAI;The Psychometrics Centre, University of Cambridge(剑桥大学心理测量中心);Cambridge Judge Business School(剑桥大学嘉治商学院)
Augmenting Rating-Scale Measures with Text-Derived Items Using the Information-Determined Scoring (IDS) Framework
Psychometrics Centre, Judge Business School, University of Cambridge(剑桥大学心理测量中心、剑桥大学商学院);Faculty of Law, University of Cambridge(剑桥大学法学院);School of Computer Science, University of Birmingham(伯明翰大学计算机科学学院);Faculty of Psychology, Beijing Normal University(北京师范大学心理学学院)
Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks
School of Computing and Information, University of Pittsburgh(匹兹堡大学计算与信息学院);Department of Educational Psychology, University of Illinois Urbana–Champaign(伊利诺伊大学厄巴纳-香槟分校教育心理学系);Department of Emergency Medicine, University of Pittsburgh(匹兹堡大学急诊医学系)
LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Maryland(马里兰大学);Virginia Tech(弗吉尼亚理工大学)
Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice
Euler Institute, Università della Svizzera italiana(欧拉研究所,瑞士大学);National Institute of Informatics(国家信息研究所);Brown University(布朗大学);University of Tübingen(图宾根大学);Insubria University(因斯布里亚大学)
Measuring Behavioural Signatures of Large Language Models through Psychometric Profiling
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Central China Normal University(华中师范大学);Shandong Science and Technology Press, Shandong Publishing Group(山东科学技术出版社,山东出版集团);Bielefeld University(比勒费尔德大学);South China Normal University(华南师范大学);Frankfurt Institute for Advanced Studies(法兰克福高等研究院)
Leveraging Computerized Adaptive Testing for Cost-effective Evaluation of Large Language Models in Medical Benchmarking
Institute of Medical Education, Health Science Center, Peking University(北京大学医学教育研究院、健康科学中心);School of Public Health, Peking University(北京大学公共卫生学院);Peking University Health Science Center-Chaoxing Joint Laboratory for Digital and Smart Medical(北京大学健康科学中心-超星数字与智能医疗联合实验室);Graduate School of Education, Peking University(北京大学教育学院)
Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models
Beijing Normal University(北京师范大学);Central China Normal University(华中师范大学);Key Laboratory of Adolescent Cyberpsychology and Behavior, Ministry of Education(教育部青少年网络心理与行为重点实验室);Beijing Key Laboratory of Applied Experimental Psychology(北京市应用实验心理学重点实验室);Research Center for Capacity Building in Educational Assessment and Evaluation (Beijing Higher Education Innovation Center for Philosophy and Social Sciences)(教育评价与评估能力建设研究中心(北京高等教育哲学社会科学创新中心))
PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematics
Adoption, usability and perceived clinical value of a UK AI clinical reference platform (iatroX): a mixed-methods formative evaluation of real-world usage and a 1,223-respondent user survey
Bridging Psychometric and Content Development Practices with AI: A Community-Based Workflow for Augmenting Hawaiian Language Assessments
University of Hawaiʻi at Mānoa College of Education(夏威夷大学马诺阿分校教育学院);Center Point Digital Solutions (contracted to Research Corporation of University of Hawaiʻi)(中心点数字解决方案(受夏威夷大学研究公司委托))