arXiv 2412.121442026-09-02cs.CLcs.AI

基于大语言模型的人格情境判断测试自动题目生成

Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models

Beijing Normal University（北京师范大学）；Central China Normal University（华中师范大学）；Key Laboratory of Adolescent Cyberpsychology and Behavior, Ministry of Education（教育部青少年网络心理与行为重点实验室）；Beijing Key Laboratory of Applied Experimental Psychology（北京市应用实验心理学重点实验室）；Research Center for Capacity Building in Educational Assessment and Evaluation (Beijing Higher Education Innovation Center for Philosophy and Social Sciences)（教育评价与评估能力建设研究中心（北京高等教育哲学社会科学创新中心））

Chang-Jin Li, Jiyuan Zhang, Yun Tang, Jian Li

备注：: Published in Computers in Human Behavior Reports. The supplementary materials, data, and items are available at https://osf.io/jbvq7/
发表信息：: Li, C.-J., Zhang, J., Tang, Y., & Li, J. (2026). Automatic item generation for personality situational judgment tests with large language models. Computers in Human Behavior Reports, 21, 100964
DOI：: 10.1016/j.chbr.2026.100964

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究利用大语言模型开发自动生成人格情境判断测试的框架，通过三项实验验证了其在内容效度、可重复性和心理测量特性上的有效性。

摘要：

人格评估通过情境判断测试（SJTs）相比传统的利克特型自我报告量表具有独特优势，但其开发仍然耗时耗力且严重依赖领域专家。最近大语言模型（LLMs）的进步显示了自动题目生成（AIG）的潜力。基于这些进展，本研究聚焦于开发和评估一个结构化且可推广的框架，用于自动生成人格SJTs，使用GPT-4和ChatGPT-5作为实证例子。进行了三项研究。研究1系统比较了提示设计和温度设置对LLM生成题目内容效度的影响，以开发一种有效的、稳定的LLM基于AIG方法。结果显示，优化的提示和温度设置为1.0在GPT-4上达到了最佳的创造力和准确性平衡。研究2通过多轮测试检验了这种自动化SJTs生成方法的跨模型通用性和可重复性。结果显示，该方法在ChatGPT-5上始终产生可重复的高质量题目。研究3评估了覆盖大五人格特质五个方面的LLM生成SJTs的心理测量特性。结果显示，大多数方面都表现出满意的可靠性与有效性，但收敛效度在顺从方面和某些方面标准效度存在局限。这些发现提供了有力证据，证明所提出的LLM基于AIG方法可以产生文化上合适且心理测量上可靠的SJTs，其效率可与或超过传统方法。

英文摘要：

Personality assessment through situational judgment tests (SJTs) offers unique advantages over traditional Likert-type self-report scales, yet their development remains labor-intensive, time-consuming, and heavily dependent on subject matter experts. Recent advances in large language models (LLMs) have shown promise for automatic item generation (AIG). Building on these developments, the present study focuses on developing and evaluating a structured and generalizable framework for automatically generating personality SJTs, using GPT-4 and ChatGPT-5 as empirical examples. Three studies were conducted. Study 1 systematically compared the effects of prompt design and temperature settings on the content validity of LLM-generated items to develop an effective and stable LLM-based AIG approach for personality SJT. Results showed that optimized prompts and a temperature of 1.0 achieved the best balance of creativity and accuracy on GPT-4. Study 2 examined the cross-model generalizability and reproducibility of this automated SJT generation approach through multiple rounds. The results showed that the approach consistently produced reproducible and high-quality items on ChatGPT-5. Study 3 evaluated the psychometric properties of LLM-generated SJTs covering five facets of the Big Five personality traits. Results demonstrated satisfactory reliability and validity across most facets, though limitations were observed in the convergent validity of the compliance facet and certain aspects of criterion-related validity. These findings provide robust evidence that the proposed LLM-based AIG approach can produce culturally appropriate and psychometrically sound SJTs with efficiency comparable to or exceeding traditional methods.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。