本文探讨大型语言模型（LLM）在评估心理测量工具内容效度方面的应用，重点关注 Big Five Questionnaire（BFQ）和 Big Five Inventory（BFI）。内容效度是测验编制的基石，用于确保心理测量充分覆盖其预期构念。我们同时利用人类专家评估和先进 LLM，比较了题项与构念语义对齐的准确性。心理学研究生采用 Content Validity Ratio（CVR）对测验题项进行评分，构成人类基线。与此同时，包括多语言模型和微调模型在内的最先进 LLM 分析题项嵌入，以预测构念映射。结果揭示了人类方法与 AI 方法各自不同的优势和局限。人类验证者擅长对齐行为信息丰富的 BFQ 题项，而 LLM 在语言简洁的 BFI 题项上表现更好。训练策略显著影响 LLM 性能，为词汇关系定制的模型优于通用 LLM。本文强调了融合人类专业知识与 AI 精确性的混合验证系统所具有的互补潜力。研究结果凸显了 LLM 在心理评估中的变革性作用，为可扩展、客观且稳健的测验开发方法铺平道路。
英文摘要：
In this article we explore the application of Large Language Models (LLMs) in assessing the content validity of psychometric instruments, focusing on the Big Five Questionnaire (BFQ) and Big Five Inventory (BFI). Content validity, a cornerstone of test construction, ensures that psychological measures adequately cover their intended constructs. Using both human expert evaluations and advanced LLMs, we compared the accuracy of semantic item-construct alignment. Graduate psychology students employed the Content Validity Ratio (CVR) to rate test items, forming the human baseline. In parallel, state-of-the-art LLMs, including multilingual and fine-tuned models, analyzed item embeddings to predict construct mappings. The results reveal distinct strengths and limitations of human and AI approaches. Human validators excelled in aligning the behaviorally rich BFQ items, while LLMs performed better with the linguistically concise BFI items. Training strategies significantly influenced LLM performance, with models tailored for lexical relationships outperforming general-purpose LLMs. Here we highlights the complementary potential of hybrid validation systems that integrate human expertise and AI precision. The findings underscore the transformative role of LLMs in psychological assessment, paving the way for scalable, objective, and robust test development methodologies.
Challenging the Validity of Personality Tests for Large Language Models
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tübingen AI Center（蒂宾根人工智能中心）；ETH Zurich（苏黎世联邦理工学院）；Methods Center（方法中心）；University of Tübingen（蒂宾根大学）
Beyond BFI: The CSI for Enhanced Reliability and Validity in Evaluating LLM Personality Traits
University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Microsoft Research Asia（微软亚洲研究院）；North Carolina State University（北卡罗来纳州立大学）
Generative Personality Simulation via Theory-Informed Structured Interview
Department of Psychological Sciences & 2 Department of Computer Science, Rice University（1 心理学系 & 2 计算机科学系，莱斯大学）；Department of Electrical and Computer Engineering, Northeastern University（3 电气与计算机工程系，东北大学）；Department of Computer Science, Johns Hopkins University（4 计算机科学系，约翰霍普金斯大学）；Microsoft Research Asia（微软亚洲研究院）
Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models
Beijing Normal University（北京师范大学）；Central China Normal University（华中师范大学）；Key Laboratory of Adolescent Cyberpsychology and Behavior, Ministry of Education（教育部青少年网络心理与行为重点实验室）；Beijing Key Laboratory of Applied Experimental Psychology（北京市应用实验心理学重点实验室）；Research Center for Capacity Building in Educational Assessment and Evaluation (Beijing Higher Education Innovation Center for Philosophy and Social Sciences)（教育评价与评估能力建设研究中心（北京高等教育哲学社会科学创新中心））
Predicting the Big Five Personality Traits in Chinese Counselling Dialogues Using Large Language Models
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西湖大学工程学院）；Westlake University（西湖大学）；School of Early Childhood Education, Hangzhou Polytechnic（杭州职业技术学院幼儿教育学院）
Computational Phenomenology of Borderline Personality Disorder: A Comparative Evaluation of LLM-Simulated Expert Personas and Human Clinical Experts
IDEAS Research Institute（IDEAS研究机构）；Adam Mickiewicz University（亚当·密茨凯维奇大学）；AMU Center for Artificial Intelligence（AMU人工智能中心）；Poznań University of Medical Sciences（波兹南医科大学）；Maria Curie-Skłodowska University（玛丽·居里-斯洛多夫斯卡大学）；University of Warsaw（华沙大学）
Cognitive phantoms in LLMs through the lens of latent variables
Department of Methodology and Statistics（方法学与统计学系）；Tilburg University（蒂尔堡大学）；Department of Security and Crime Science（安全与犯罪科学系）；University College London（伦敦大学学院）
Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Need
Department of Computer；Information Engineering, Hebei Petroleum University of Technology, Chengde, China (e-mail；School of Computer Science \& Technology, Beijing Institute of Technology, Beijing, China (e-mail
Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side Verification
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition（脑启发式感知与认知MoE实验室）；University of Science and Technology of China（中国科学技术大学）；Noah's Ark Lab, Huawei Technologies（华为技术诺亚实验室）；Tianjin University（天津大学）
The use of GPT-4o and Other Large Language Models for the Improvement and Design of Self-Assessment Scales for Measurement of Interpersonal Communication Skills
University of Zagreb（萨格勒布大学）；Faculty of Organization and Informatics（组织与信息学院）
Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact
Max Planck Institute for Human Development（马克斯·普朗克人类发展研究所）；University of Konstanz（康斯坦茨大学）；Barcelona Supercomputing Center（巴塞罗那超级计算中心）；University of Basel（巴塞尔大学）
Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead
Max Planck Institute for Intelligent Systems, Tübingen（马克斯·普朗克智能系统研究所，图宾根）；ETH Zurich, Zurich（苏黎世联邦理工学院，苏黎世）；University of Tübingen, Methods Center, Tübingen（图宾根大学，方法中心，图宾根）；Massachusetts Institute of Technology, Cambridge（麻省理工学院，剑桥）
